To avoid being asked about removing files, add the -f (“force”) option:

rm -f /path/to/file