The nearby A4-type star Fomalhaut hosts a debris belt in the form of an eccentric ring, which is thought to be caused by dynamical influence from a giant planet companion. In 2008, a detection of a point source inside the inner edge of the ring was reported and was interpreted as a direct image of the planet, named Fomalhaut b. The detection was made at 600-800nm, but no corresponding signatures were found in the near-infrared range, where the bulk emission of such a planet should be expected. Here, we present deep observations of Fomalhaut with Spitzer/IRAC at 4.5 µm, using a novel point-spread function subtraction technique based on angular differential imaging and Locally Optimized Combination of Images, in order to substantially improve the Spitzer contrast at small separations. The results provide more than an order ofmagnitude improvement in the upper flux limit of Fomalhaut b and exclude the possibility that any flux from a giant planet surface contributes to the observed flux at visible wavelengths. This renders any direct connection between the observed light source and the dynamically inferred giant planet highly unlikely. We discuss several possible interpretations of the total body of observations of the Fomalhaut system and find that the interpretation that best matches the available data for the observed source is scattered light from a transient or semi-transient dust cloud.