John Mueller from Google said on Reddit that if your cache link in Google returns a 404 error, that it doesn’t mean your rankings will be hurt in Google. John said “that can happen, it doesn’t mean anything.”

So if you are worried that this might result in your page being de-indexed or the rankings may drop, Google says no, no need to worry about that. The Google Cache 404 error is an internal issue with the Google Cache server and doesn’t reflect on indexing or ranking of the web page.