The present study aimed to test whether the Time is Space conceptual metaphor is grounded on the sensori-motor experience of reading and writing. Two groups of participants differing in their directional reading/writing system (Spanish and Hebrew) carried out a time judgment task on auditorily presented words referring either to the past or the future. As expected, our results showed opposite congruency effects in the two groups: Spanish participants were faster responding to past words with their left hand and to future words with their right hand, whereas Hebrew participants showed the opposite pattern. However, contrary to Santiago et al. (2007) with visual stimuli, we did not observe a facilitation effect at the perceptual level: stimulus location (left/right) did not interact with temporal meaning. In two additional experiments, the saliency of the auditory spatial frame of reference was increased, relative to the visual spatial frame, by asking Spanish participants to perform the task blind-folded. Under these conditions, temporal meaning interacted with both stimulus location and response side.