It seems I’m not the only one who’s puzzled. The Guardian featured this question in their Notes & Queries section (Why are dusters yellow? Notes & Queries, The Guardian)
I imagined that there’d be a simple answer, but there doesn’t seem to be a definitive one. The one I find most convincing is the one about dusters once being recycled butter muslin, which absorbed the colouring used in the butter it once wrapped.