How do you know when it's time to finally walk away from a prospect?

When it is not a win-win for both. 

It is important to objectively determine if you are able to drive any value for the client and does the client recognize/ accept that value proposition. 

As an example, a managed service prospect solely expecting a reduced price but unwilling to adapt to new mature ways of working  is not a good prospect to go after. 

HIgh performing sales teams have a consistent and easy-to-inspect qualification model.  It should be based on the buyer journey rather than the sales persons activities.  When that model maps to the buyer's journey and reflects the buying process and timeframe, walking away from an unwinnable opportunity becomes an easy decision.