Do you have a rental agreement with them? If so, do you have a refund policy section in that agreement that explains that no refunds will be given in case of disputes? I am not a lawyer, but I would think that it will be very difficult for the renters to get their money back in court, unless they have clear proof (pictures) that your vacation rental was very dirty. Especially that you offered to solve the issue as soon as possible and in a reasonable way.
However, you may want to avoid any negative publicity that this could cause for your vacation rental and try to solve this by finding a reasonable resolution for both of you (may be just keeping their deposit and refunding them the rest?).
Hopefully some other vacation rental owners that have been in such a situation could give you some real life experiences on this topic.