1. Dust specs would respond to momentum imparted by the light much more quickly, which would reduce the momentum difference between the ship and the specs.

2. Enough incident light could vaporize the specs entirely leaving only gas or plasma, which doesn't carry the same danger to the ship.