A large genotyping project within the Breast Cancer Association Consortium (BCAC) recently identified 41 associations between single nucleotide polymorphisms (SNPs) and overall breast cancer (BC) risk. We investigated whether the effects of these 41 SNPs, as well as six SNPs associated with estrogen receptor (ER) negative BC risk are modified by 13 environmental risk factors for BC. Data from 22 studies participating in BCAC were pooled, comprising up to 26,633 cases and 30,119 controls. Interactions between SNPs and environmental factors were evaluated using an empirical Bayes-type shrinkage estimator. Six SNPs showed interactions with associated p-values (pint) <1.1 × 10−3. None of the observed interactions was significant after accounting for multiple testing. The Bayesian False Discovery Probability was used to rank the findings, which indicated three interactions as being noteworthy at 1% prior probability of interaction. SNP rs6828523 was associated with increased ER-negative BC risk in women ≥170 cm (OR = 1.22, p = 0.017), but inversely associated with ER-negative BC risk in women <160 cm (OR = 0.83, p = 0.039, pint = 1.9 × 10−4). The inverse association between rs4808801 and overall BC risk was stronger for women who had had four or more pregnancies (OR = 0.85, p = 2.0 × 10−4), and absent in women who had had just one (OR = 0.96, p = 0.19, pint = 6.1 × 10−4). SNP rs11242675 was inversely associated with overall BC risk in never/former smokers (OR = 0.93, p = 2.8 × 10−5), but no association was observed in current smokers (OR = 1.07, p = 0.14, pint = 3.4 × 10−4). In conclusion, recently identified BC susceptibility loci are not strongly modified by established risk factors and the observed potential interactions require confirmation in independent studies.