Europe PMC Funders Group Author Manuscript J Neurosci. Author manuscript; available in PMC 2010 October 19. Published in final edited form as: J Neurosci. 2007 October 17; 27(42): 11431–11441. doi:10.1523/JNEUROSCI.2252-07.2007.
Europe PMC Funders Author Manuscripts
Audio-visual temporal correspondence modulates multisensory superior temporal sulcus plus primary sensory cortices Toemme Noesselt1,2, Jochem W. Rieger2, Mircea Ariel Schoenfeld2, Martin Kanowski2, Hermann Hinrichs2, Hans-Jochen Heinze2, and Jon Driver1 1UCL Institute of Cognitive Neuroscience and Department of Psychology, University College London, 17 Queen Square, London WC1N 3AR, UK 2Department
of Neurology II & Center for Advanced Imaging, Otto-von-Guericke-Universität, Leipziger Str.44, 39120 Magdeburg, Germany
Summary
Europe PMC Funders Author Manuscripts
The brain should integrate related but not unrelated information from different senses. Temporal patterning of inputs to different modalities may provide critical information about whether those inputs are related or not. We studied effects of temporal correspondence between auditory and visual streams on human brain activity with fMRI. Streams of visual flashes with irregularly jittered timing (mean rate 4Hz) could appear on the right or left, with or without a stream of auditory tones that either coincided perfectly when present (highly unlikely by chance); or were non-coincident with vision (different erratic pattern with same mean rate); or an auditory stream appeared alone. fMRI revealed BOLD-increases in multisensory superior temporal sulcus (mSTS), contralateral to a visual stream when coincident with an auditory stream, and BOLD-decreases for non-coincidence relative to unisensory baselines. Contralateral primary visual cortex and auditory cortex were also activated by audio-visual temporal correspondence, as confirmed in individuals. Connectivity analyses indicated enhanced influence from mSTS upon primary sensory areas, rather than vice-versa, during audio-visual correspondence. Temporal correspondence between auditory and visual streams affects a network of both multisensory (mSTS) and sensory-specific areas, including even primary visual and auditory cortex, with stronger responses for corresponding and thus related audio-visual inputs.
Keywords multisensory; audiovisual; temporal correspondence; fMRI; functional connectivity
Introduction Among the many signals entering our senses, some inputs to one sense (e.g. audition) may relate temporally and/or spatially to inputs entering another sense (e.g. vision) when they originate from the same object. Ideally the brain should integrate just those multisensory inputs that reflect a common external source, as may be indicated by spatial, temporal or semantic constraints (Stein and Meredith, 1993; Calvert et al., 2004; Spence and Driver, 2004; Macaluso and Driver, 2005; Schroeder and Foxe, 2005). Many neuroscience and human neuroimaging studies have investigated possible spatial constraints on multisensory integration (e.g. Wallace et al., 1996; Macaluso et al., 2000; McDonald et al., 2000;
Correspondence to: Dr. Toemme Noesselt, Center for Advanced Imaging, Haus 1, Leipziger Str.44, 39120 Magdeburg, Germany Tel: (+49)-(0)391-6713430 Fax: (+49)-(0)391-6715233 Email:
[email protected].
Noesselt et al.
Page 2
McDonald et al., 2003; Macaluso et al., 2004); or factors that may be more ‘semantic’ (e.g. for integration of matching speech-sounds and lip-movements (Calvert et al., 1997), or of visual objects with matching environmental sounds (Beauchamp et al., 2004a, 2004b; Beauchamp, 2005a).
Europe PMC Funders Author Manuscripts
Here we focus on possible constraints from temporal correspondence only (see Stein et al., 1993; Calvert et al., 2001; Bischoff et al., 2007; Dhamala et al., 2007). We used streams of non-semantic stimuli (visual transients and beeps) to isolate purely temporal influences. We arranged that audio-visual temporal relations should convey strong information that auditory and visual streams were related, or unrelated, by using erratic rapid temporal patterns which either matched perfectly between audition and vision (very unlikely by chance) or mismatched substantially, but with the same average rate. We anticipated increased brain activations for temporally-coincident audio-visual streams (as compared with noncoincident or unisensory streams, see Methods) in multisensory superior temporal sulcus (mSTS). This region is known to receive converging auditory and visual inputs (Kaas and Collins, 2004). Moreover, mSTS is thought to play some role(s) in multisensory integration (Benevento et al., 1977; Bruce et al., 1981; Cusick, 1997; Beauchamp et al., 2004b), and was influenced by audio-visual synchrony in some prior fMRI studies that used very different designs than here and/or more semantic stimuli (e.g. Calvert et al., 2001; Atteveldt et al., 2006; Bischoff et al., 2007; Dhamala et al., 2007).
Europe PMC Funders Author Manuscripts
There have been several recent proposals that multisensory interactions may affect not only established multisensory brain regions (such as mSTS), but also brain areas (or evoked responses) traditionally considered sensory-specific (e.g. Brosch and Scheich, 2005; Foxe and Schroeder, 2005 for reviews), though some past ERP-examples had proved somewhat controversial (see Teder-Sälejärvi et al., 2002). Here we sought to test directly with fMRI whether audio-visual correspondence in purely temporal patterning might affect (contralateral) sensory-specific visual and auditory cortices. This may be expected if temporal coincidence can render visual and auditory stimuli more salient or perceptually intense, as suggested by some psychophysical work (Stein et al., 1996; Frassinetti et al., 2002; Lovelace et al., 2003). Here we provide an unequivocal demonstration that audiovisual correspondence in temporal patterning can indeed affect even primary visual and auditory cortex (V1 and A1), as well as contralateral mSTS.
Methods Twenty four neurologically normal subjects (10 female, mean age 24) participated after written informed consent in accord with local ethics. Visual stimulation was in the upper left hemifield for 12 subjects, in the upper right for the other 12. This was presented at the top of the MR-bore via clusters of 4 optic fibres arranged into a rectangular shape, and 5 interleaved fibres arranged into a cross shape, 20 above the horizontal meridian at an eccentricity of 180. Visual stimuli were presented peripherally, which may maximise the opportunity for interplay between auditory and visual cortex (see Falchier et al., 2002), and also allowed us to test for any contralaterality in effects for one visual field or the other. The peripheral fibre-optic endings could be illuminated red or green with a standard luminance of 40 cd/m2 and were 1.50 in diameter (see Fig 1c for schematics of the resulting colored ‘shapes’). Streams of visual transients were produced by switching between the differently colored cross and rectangle shapes (red and green respectively in Figure 1c, but shape-color was counterbalanced across subjects). Throughout each experimental run, subjects fixated a central fixation cross of ~0.20 in diameter. Eight red-green (cross/square) reversals occurred in a 2 second interval, with the SOA between each successive color-change ranging in a pseudorandom fashion from 100 to 500 ms (mean reversal rate of 4 Hz, with rectangular distribution from 2 to 10 Hz, but note that reversal rate was never constant for successive
J Neurosci. Author manuscript; available in PMC 2010 October 19.
Noesselt et al.
Page 3
Europe PMC Funders Author Manuscripts
transients), to produce a uniquely jittered timing for each 2 sec segment. Auditory stimuli were presented via a piezo-electric speaker inside the scanner, just above fixation. Each auditory stimulus was a clearly audible 1 kHz sound-burst with duration of 10 ms at ~70 dB. Identical temporally-jittered stimulation sequences within vision and/or audition were used in all conditions overall (fully counterbalanced), so that there was no difference whatsoever in temporal statistics between conditions, except for the critical temporal relation between auditory and visual streams during multisensory trials (unisensory conditions were also included, see below). The experimental stimuli (for the visual-only baseline, auditory-only baseline, and for audiovisual temporal correspondence (AVC) or non-correspondence (NC)) were all presented during silent periods (2 s) interleaved with scanning (3 s periods of fMRI acquisition) to prevent scanner-noise interfering with our auditory stimuli or perception of their temporal relation with visual flashes. In the AVC condition, a tone burst was initiated synchronously with every visual transient (see Fig 1a) and thus had exactly the same pseudorandom temporal pattern. During the NC condition (Fig 1b), tone bursts occurred with a different pseudo-random temporal pattern (but always having the same overall temporal statistics, including mean rate of 4 Hz within a rectangular distribution from 2 to 10 Hz), with a minimal protective ‘window’ of 100 ms now separating each sound from onset of a visual pattern-reversal (Fig 1b).
Europe PMC Funders Author Manuscripts
This provided clear information that the two streams were either strongly related, as in the AVC condition (such perfect coincidence for the erratic temporal patterns is exceptionally unlikely to arise by chance); or were unrelated, as for the NC condition. During the latter non-coincidence, up to two events in one stream could occur before an event in the second stream had to occur. The mean 4 Hz stimulation rate used here, together with the constraints (protective window, see Fig 1b) implemented to avoid any accidental synchronies in the non-corresponding condition, should optimise detection of audio-visual correspondence versus non-correspondence (see Fujisaki et al., 2006), while making these bimodal conditions otherwise identical in terms of the temporal patterns presented overall to each modality. All sequences were created individually for each subject using Matlab 6.5. Piloting confirmed that the correspondence versus non-correspondence relation could be discriminated readily when requested (mean percent correct 93.8%), even with such peripheral visual stimuli. Irregular stimulus trains were chosen, as this makes an audiovisual temporal relation much less likely to arise by chance alone, and hence (a)sychrony typically becomes easier to detect than for regular frequencies, or for single auditory and visual events rather than stimulus trains (see also Slutsky and Recanzone, 2001; Noesselt et al., 2005). Two unisensory conditions (i.e. visual or auditory streams alone) were also run. These allowed our fMRI analysis to distinguish candidate multisensory brain regions (responding to either type of unisensory stream) from sensory-specific regions (visually- or auditorilyselective); see below. Throughout each experimental run, participants performed a central visual monitoring task requiring detection of occasional brief (1 ms) brightening of the fixation point via button press. This could occur at random times (average rate 0.1 Hz) during both stimulation and scan periods. Participants were instructed to perform this fixation-monitoring task, and that auditory and peripheral visual stimuli were task-irrelevant. We chose this fixationmonitoring task to avoid the different multisensory conditions being associated with changes in performance that might otherwise have contaminated the fMRI data; because we were interested in stimulus-determined (rather than task-determined) effects of audio-visual
J Neurosci. Author manuscript; available in PMC 2010 October 19.
Noesselt et al.
Page 4
temporal correspondence; and so as to minimize eye movements. Eye-position was monitored online during scanning (Kanowski et al., 2007).
Europe PMC Funders Author Manuscripts
fMRI data were collected in 4 runs with a neuro-optimized 1.5 GE scanner equipped with a head-spine-coil. A rapid sparse-sampling protocol was used (136 volumes per run with 30 slices covering whole brain; TR of 3s; Silent Pause of 2s; TE of 40 ms; Flip angle of 90; resolution of 3.5×3.5 mm; 4 mm slice thickness; FOV was 20 cm). Experimental stimuli were presented during the silent scanner periods (2 s scanner pauses). Each mini-block lasted 20 s per condition, containing 8 s (4 × 2) of stimulation (with each successive 2 s segment of stimuli then separated by 3 s of scanning). These mini-blocks of experimental stimulation in one of the four conditions or another (random sequence) were each separated by 20 s blocks, in which only the central fixation task was presented (unstimulated blocks). After pre-processing for motion correction, normalisation, and 6mm smoothing, data were analysed in SPM2 by modelling the 4 conditions and the intervening unstimulated baselines with box-car functions. Voxel-based group-effects were assessed with a second-level random-effects analysis, identifying candidate multisensory regions (responding to both auditory and visual stimulation); sensory-specific regions (difference between visual minus auditory, or vice-versa); and the critical differential effects of coincident minus noncoincident audio-visual presentations.
Europe PMC Funders Author Manuscripts
Conjunction analyses assessed activation within sensory-specific and multisensory cortex (thresholded at p .2), with maintenance of central fixation also equally good across conditions (i.e. similar performance for all conditions (less than 2° deviation in 98% of trials)), as expected given the task at central fixation. Modulation of BOLD-responses due to audiovisual correspondence
Europe PMC Funders Author Manuscripts
For fMRI analyses, the random-effect SPM analysis confirmed that unisensory visual streams activated sensory-specific occipital visual cortex; while auditory streams activated auditory core, belt and parabelt regions in temporal cortex (see Table 1). Candidate multisensory regions, activated by both the unisensory visual and unisensory auditory streams, included bilateral posterior STS, posterior parietal and dorsolateral prefrontal areas. However, within these candidate multisensory regions only STS showed the critical effects of audio-visual temporal correspondence (see Tab 2a and Figure 2a). Within the functionally-defined multisensory regions, audio-visual temporal correspondence (AVC) minus non-correspondence (NC) specifically activated (at p non-coincident condition
0,001
0,001
P<
Peak T
y
-88
MNI-coordinates x
-26
-74
Brain region
0,001
-30
Contralateral right hemisphere
Visual Stimuli in LVF
Transversal occipital sulcus
0,001
-76
4,70
Fusiform gyrus
-8
3,01
Lingual gyrus/calcarine
0,001
y
x
MNI-coordinates P<
Peak T
Brain region
Contralateral left hemisphere
Visual Stimuli in RVF
2B. Visual areas: Audiovisual coincident > non-coincident condition
-48
3,19
Posterior STS
60
y
0,001
MNI-coordinates
-50
Peak T
x
-54
Brain region P<
0,001
3,43
Contralateral right hemisphere
Visual Stimuli in LVF
Posterior STS
y
x
MNI-coordinates P<
Peak T
Brain region
Contralateral left hemisphere
Visual Stimuli in RVF
2 A. Multisensory areas: Audiovisual coincident > non-coincident condition
36
-4
10
z
30
-16
2
z
12
z
8
z
Group average activation-peaks for the experimental contrast audiovisual Coincidence > Non-Coincidence within multisensory or sensory-specific regions (i.e. significant effect of visual minus auditory stimulation, or vice-versa). Only clusters containing more than 20 voxels are described (see Methods and Figure 2).
Europe PMC Funders Author Manuscripts Tab 2 Noesselt et al. Page 24
Europe PMC Funders Author Manuscripts 0,005
2,95
0,001
4,19 3,89
0,001 0,001
3,46 3,86
4,18 3,55
Planum temporale Middle STG
Left Hemisphere
0,001
0,001
0,001
3,46
Heschl’s gyrus Planum temporale/STS Planum polare
P<
Peak T
Right hemisphere
Brain region
Visual Stimuli in LVF
Middle STS
0,001
3,89
Middle STG
0,001
0,001
3,03
Planum polare
Right Hemisphere
Planum polare
Middle STG
64
38
-40
54
48
x
50
56
54
-48
-66
-34
-36
-22
-28
-20
y
MNI-coordinates
-10
-18
-4
-14
-28
14
18
-4
6
10
z
-4
2
-8
-6
12
2C. Auditory areas: Audiovisual coincident > non-coincident condition
Noesselt et al. Page 25
Europe PMC Funders Author Manuscripts
J Neurosci. Author manuscript; available in PMC 2010 October 19.
