A synesthetic metaphor (e.g., sweet touch) is a metaphor that results from a combination of a modifier and a head, where both express different perceptual qualities. Most of the existing studies examine how the acceptability of synesthetic metaphors can be explained by the pairing of adjective modifierfs and head nounfs modalities. However, little attention has been given to cognitive effects evoked by synesthetic metaphors. This paper explores cognitive effects evoked by synesthetic metaphors for the Japanese language. Based on Abstract Performance Grammar (APG) proposed by Osgood (1980) we analyzed how semantic interactions between vehicle and topic of synesthetic metaphors function to shift the meanings of words to the positive pole or negative pole. In the experiment 3267 subjects were asked to evaluate meanings of 158 linguistic expressions by 7-point semantic differential scales. Results show that synesthetic metaphors, especially synesthetic metaphors modified by color adjectives, tend to evoke negative effects, which is against the rules predicted by APG model.