To explain the properties of dust in the interstellar medium (ISM), the presence of a refractory organic mantle is necessary. The outflows of AGB stars are among the main contributors of stellar dust to the ISM. We present the first study of the refractory organic contribution of AGB stars to the ISM. Based on laboratory experiments, we included a new reaction in our extended chemical kinetics model: the photoprocessing of volatile complex ices into inert refractory organic material. The refractory organic feedback of AGB outflows to the ISM is estimated using observationally motivated parent species and grids of models of C-rich and O-rich outflows. Refractory organic material is mainly inherited from the gas phase through accretion onto the dust and subsequent photoprocessing. Grain-surface chemistry, initiated by photodissociation of ices, produces only a minor part and takes place in a sub-monolayer regime in almost all outflows. The formation of refractory organic material increases with outflow density and depends on the initial gas-phase composition. While O-rich dust is negligibly covered by refractory organics, C-rich dust has an average coverage of 3 − 9%, but can be as high as 8 − 22%. Although C-rich dust does not enter the ISM bare, its average coverage is too low to influence its evolution in the ISM or significantly contribute to the coverage of interstellar dust. This study opens up questions on the coverage of other dust-producing environments. It highlights the need for an improved understanding of dust formation and for models specific to density structures within the outflow.