Support `304` and non-modified content better

[navarone-feekery]

Problem Description

This came up in #219

We currently don't support checking for modified content, but we could do this. If a server returns a 304 we just drop updating the content. Because 304 responses don't contain any content, we can't re-index the content from that URL as part of the response processing. This means that if a server has previously ingested a page when it returned 200, but that page now returns 304, the page will be deleted from the index and the content becomes invisible to Crawler.

Proposed Solution

Crawler should treat 304 responses in the following way:

• Save URLs for 304 in a temporary list (in-memory)
• During purge crawl phase, if the URL is related to content from already indexed into ES, don't delete the doc
  • Instead, update last_crawled_at value
• Output this information in debug logger