The smooth pursuit system moves the eyes in space accurately while compensating for visual inputs from the moving background and/or vestibular inputs during head movements. To understand the mechanisms underlying such interactions, we examined the influence of a stationary textured visual background on smooth pursuit tracking and compared the results in young and adult humans and monkeys. Six humans (three children, three adults) and six macaque monkeys (five young, one adult) were used. Human eye movements were recorded using infrared oculography and evoked by a sinusoidally moving target presented on a computer monitor. Scleral search coils were used for monkeys while they tracked a target presented on a tangent screen. The target moved in a sinusoidal or trapezoidal fashion with or without whole body rotation in the same plane. Two kinds of backgrounds, homogeneous and stationary textured, were used. Eye velocity gains (eye velocity/target velocity) were calculated in each condition to compare the influence of the textured background. Children showed asymmetric eye movements during vertical pursuit across the textured (but not the homogeneous) background; upward pursuit was severely impaired, and consisted mostly of catch-up saccades. In contrast, adults showed no asymmetry during pursuit across the different backgrounds. Monkeys behaved similarly; only slight effects were observed with the textured background in a mature monkey, whereas upward pursuit was severely impaired in young monkeys. In addition, VOR cancellation was severely impaired during upward eye and head movements, resulting in residual downward VOR in young monkeys. From these results, we conclude that the directional asymmetry observed in young primates may reflect a different neural organization of the vertical, particularly upward, pursuit system in the face of conflicting visual and vestibular inputs that can be associated with pursuit eye movements. Apparently, proper compensation matures later.