Don't be dramatic

Maybe they quietly return at night to move out in the morning

You can ride at night if there is moon or starlight and in full darkness if the horse knows the route