Skip to content

Commit 2c2d1f1

Browse files
committed
serialize values in KeyValueStore the same way for every backend
1 parent fddc577 commit 2c2d1f1

34 files changed

Lines changed: 1840 additions & 1069 deletions

‎docs/03_concepts/03_crawling_context.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ The following example uses most of these helpers in a <ApiLink to="class/ParselC
5656

5757
## Results are committed after the handler succeeds
5858

59-
The `push_data`, `add_requests` and `enqueue_links` calls and the `set_value` calls on a key-value store from `get_key_value_store` don't write anything right away. The context records them in an internal `RequestHandlerRunResult` object, and the crawler commits the recorded changes only after the request handler returns without an exception. Reading a key from that key-value store in the same handler returns the value you've set, even before the commit.
59+
The `push_data`, `add_requests` and `enqueue_links` calls and the `set_value` calls on a key-value store from `get_key_value_store` don't write anything right away. The context records them in an internal `RequestHandlerRunResult` object, and the crawler commits the recorded changes only after the request handler returns without an exception. The key-value store keeps each value as it was when you called `set_value`, so changing the object afterwards doesn't change what's committed. Reading a key from that key-value store in the same handler returns the value you've set, even before the commit. It comes back the way the store reads it after the commit, so a tuple is read as a list.
6060

6161
When the handler raises, the crawler discards the recorded changes and retries the request. A handler that pushes an item and then fails on the next line doesn't leave a partial or duplicate item in the dataset. The retry starts with an empty record and produces the output once, when it succeeds. The same applies to requests added by a failed attempt.
6262

‎docs/03_concepts/12_storage_backends.mdx‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -460,6 +460,7 @@ class RedisKeyValueStoreBackend {
460460
class KeyValueStoreKeys {
461461
key_value_stores:[name]:items - Hash
462462
key_value_stores:[name]:metadata_items - Hash
463+
key_value_stores:[name]:keys - Sorted Set
463464
key_value_stores:[name]:metadata - JSON Object
464465
}
465466

‎docs/03_concepts/26_extending_crawlee.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ A storage backend is the layer behind Crawlee's three storages. <ApiLink to="cla
148148
The <ApiLink to="class/StorageBackend">`StorageBackend`</ApiLink> contract defines three factory methods: `create_dataset_backend`, `create_kvs_backend`, and `create_rq_backend`. The returned backends define the rest of the contract:
149149

150150
- <ApiLink to="class/DatasetBackend">`DatasetBackend`</ApiLink> handles appending and reading items.
151-
- <ApiLink to="class/KeyValueStoreBackend">`KeyValueStoreBackend`</ApiLink> handles record access and iteration.
151+
- <ApiLink to="class/KeyValueStoreBackend">`KeyValueStoreBackend`</ApiLink> handles reading and writing records as bytes and listing their keys a page at a time.
152152
- <ApiLink to="class/RequestQueueBackend">`RequestQueueBackend`</ApiLink> handles adding, fetching, and marking requests as handled.
153153

154154
A custom backend implements all four classes.

‎docs/03_concepts/code_examples/storages/kvs_basic_example.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ async def main() -> None:
1212
await kvs.set_value(key='some-key', value={'foo': 'bar'})
1313

1414
# Get the value associated with 'some-key'.
15-
value = kvs.get_value('some-key')
15+
value = await kvs.get_value('some-key')
1616
# Do something with it...
1717

1818
# Delete the value associated with 'some-key' by setting it to None.

‎docs/06_upgrading/upgrading_to_v2.md‎

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -176,9 +176,64 @@ Data written by v1 still loads, so an existing storage directory, Redis instance
176176

177177
## `MemoryStorageBackend` stores copies
178178

179-
`MemoryStorageBackend` stores a JSON snapshot of each dataset item and of each key-value store value with a JSON content type, serialized the same way as by `FileSystemStorageBackend`. Changing an object after you push or set it doesn't change the stored data, and pushing the same object twice stores two independent items. Reads return fresh copies too, so changing a returned item or value doesn't change the store.
179+
`MemoryStorageBackend` stores a JSON snapshot of each dataset item, serialized the same way as by `FileSystemStorageBackend`. Changing an object after you push it doesn't change the stored data, and pushing the same object twice stores two independent items. Reads return fresh copies too, so changing a returned item doesn't change the dataset.
180180

181-
The memory backend now handles values the same way as the file-system backend. It raises on a value the file-system backend can't serialize, such as one with a circular reference or a tuple as a dict key. Values that JSON doesn't support, such as a `datetime`, are stored as strings, and dict keys that aren't strings become strings.
181+
The memory backend now handles dataset items the same way as the file-system backend. It raises on an item the file-system backend can't serialize, such as one with a circular reference or a tuple as a dict key. Values that JSON doesn't support, such as a `datetime`, are stored as strings, and dict keys that aren't strings become strings.
182+
183+
Key-value store values are serialized by `KeyValueStore` itself, the same way for every storage backend. For details, see [`KeyValueStore` serializes and parses the values](#keyvaluestore-serializes-and-parses-the-values).
184+
185+
## `KeyValueStore` serializes and parses the values
186+
187+
`KeyValueStore` now serializes a value before passing it to the storage backend, and parses the stored bytes after reading them. In v1, each storage backend did both on its own. The rules are the same for every storage backend. The v1 behavior described below is that of the file-system, SQL and Redis backends. `MemoryStorageBackend` kept each value as it was set.
188+
189+
### `set_value` with `None` deletes the record
190+
191+
`set_value(key, None)` deletes the record, the same as `delete_value(key)`. In v1, it stored a record with the `application/x-none` content type, so `record_exists` returned `True` for it. A record stored that way by v1 is now read as empty `bytes`. Such records include the persisted state of `Statistics` and `SessionPool`, which v1 cleared this way when `run()` was called again after a finished run. If the process stopped before that state was saved again, the next start fails with a `ValidationError` for the record. Delete the record to start from the default state.
192+
193+
### Values without a content type
194+
195+
Without `content_type`, `bytes`, `bytearray` and `memoryview` values are stored as `application/octet-stream`, `str` values as `text/plain`, and any other value as JSON. In v1, only `dict` and `list` values were stored as JSON. Numbers and booleans were stored as text and read back as `str`, and other objects were stored as the bytes of their `str()` form.
196+
197+
```python
198+
await kvs.set_value('count', 5)
199+
200+
# v1
201+
await kvs.get_value('count') # '5'
202+
203+
# v2
204+
await kvs.get_value('count') # 5
205+
```
206+
207+
### Values with a content type
208+
209+
A `str` or bytes value is stored as it is under the given `content_type`. In v1, a value with a JSON content type was always serialized, so a string that already held JSON was encoded twice and read back as a string. In v2, it's read back parsed. A string that isn't valid JSON is stored too, and reading it raises `ValueError`. Store such a string without the JSON content type. `Dataset.export_to` with the JSON content type now writes the items as a JSON array, where v1 wrote that array wrapped in a JSON string.
210+
211+
A `str` value is encoded with the charset of its content type. A charset that Python doesn't know raises `LookupError`, and a character that the charset can't encode raises `UnicodeEncodeError`. In v1, a `str` value was always encoded as UTF-8. A `str` value that v1 stored under a content type with another charset is now decoded with that charset, so its non-ASCII characters come back wrong. To read its text, decode the bytes from the new `KeyValueStore.get_record` as UTF-8.
212+
213+
Any other value can only be stored as JSON. Passing it with a content type that isn't JSON raises `TypeError`. In v1, such a value was stored as the bytes of its `str()` form.
214+
215+
### Reading values
216+
217+
`get_value` parses a record by its content type. `application/json` is parsed, `text/*` and `application/*xml` are decoded to `str` with the charset of the content type, or with UTF-8 when it has none, and any other content type is returned as `bytes`. A JSON or text record with a charset that Python doesn't know is returned as `bytes` too. In v1, `application/*xml` content types were returned as `bytes`, and the charset was ignored.
218+
219+
A record with a JSON content type that isn't valid JSON raises `ValueError` from `get_value`, `iterate_values` and `iterate_entries`. In a text or JSON record, the bytes that its charset can't decode are replaced with the Unicode replacement character. In v1, the storage backend logged a warning in both cases, and the record was reported as missing. To read the stored bytes of such a record, use `KeyValueStore.get_record`.
220+
221+
### Values set in a request handler
222+
223+
The key-value store from `context.get_key_value_store()` serializes a value when you call `set_value`. Changes made to the object afterwards aren't stored when the crawler commits the results of the handler, and `get_value` in the same handler returns a new copy of the value. In v1, the store kept the object itself until the commit, so those changes were stored, and `get_value` returned the same object. A value that can't be stored now raises from `set_value` in the handler, where v1 raised only during the commit.
224+
225+
### Custom key-value store backends
226+
227+
A custom `KeyValueStoreBackend` no longer serializes or parses the values:
228+
229+
- `set_value(*, key, value, content_type)` receives the value as `bytes`, and `content_type` is always set.
230+
- `get_value(*, key)` returns a `KeyValueStoreRecord` whose `value` is the stored `bytes`. `KeyValueStoreRecord` is no longer generic.
231+
- `list_keys(*, prefix, exclusive_start_key, limit)` replaces `iterate_keys` and `iterate_entries`. It returns a single `KeyValueStoreKeysListPage`, whose `items` are ordered by key. Its `next_exclusive_start_key` is where the next page starts, or `None` on the last page. A page before the last one may hold fewer than `limit` keys, or none. The page also carries `count`, the number of keys on it, and the `limit` and `exclusive_start_key` it was requested with.
232+
- `get_values(*, keys)` is optional. Override it if the backend can read several records in a single call. It returns a mapping from each key that has a record to that record. The default implementation calls `get_value` for each key. The built-in file-system, SQL and Redis backends override it, so a subclass of one of them that changes `get_value` must change `get_values` the same way.
233+
234+
### `RedisStorageBackend` indexes the record keys
235+
236+
`RedisStorageBackend` keeps the keys of each key-value store in a sorted set, `key_value_stores:{name}:keys`, so it lists them a page at a time. When v2 opens a store written by v1, it builds the set from the existing records. Don't write to the same store from v1 and v2 at the same time, since v2 doesn't list the keys that v1 adds after the set exists.
182237

183238
## SQL and Redis lock fetched requests for 3 minutes
184239

@@ -688,6 +743,8 @@ Messages that mentioned a storage client name the storage backend instead. The D
688743

689744
`RequestQueue.purge()` no longer logs the `Storage client "..." does not support purging the request queue.` warnings. It raises `NotImplementedError` instead. For details, see [`RequestQueue.purge` raises when the storage backend can't purge](#requestqueuepurge-raises-when-the-storage-backend-cant-purge).
690745

746+
The storage backends no longer log the `Failed to decode JSON value for key "..."` and `Failed to decode text value for key "..."` warnings. Instead, `KeyValueStore` raises `ValueError` for a record that isn't valid JSON, and replaces the bytes it can't decode in a text or JSON record. For details, see [Reading values](#reading-values). `RedisStorageBackend` no longer logs the `Metadata for key "..." is missing or invalid.` warning when it reads a key that has no record.
747+
691748
The ERROR message the crawler logs before raising `PersistentRateLimitError` now reads "Giving up the crawl because the request manager can't make progress." instead of "Giving up the crawl because a domain keeps rate-limiting every request."
692749

693750
The two DEBUG messages that `RequestQueue.is_finished` logged, "Background add requests tasks are still in progress." and "The request queue is finished.", are removed together with the method.

‎src/crawlee/_types.py‎

Lines changed: 31 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -234,13 +234,14 @@ async def set_value(
234234

235235
@dataclass()
236236
class KeyValueStoreValue:
237-
content: Any
238-
content_type: str | None
237+
content: bytes
238+
content_type: str
239239

240240

241241
class KeyValueStoreChangeRecords:
242242
def __init__(self, actual_key_value_store: KeyValueStore) -> None:
243-
self.updates = dict[str, KeyValueStoreValue]()
243+
self.updates = dict[str, KeyValueStoreValue | None]()
244+
"""The pending value of each key, serialized as it will be stored, or `None` to delete the record."""
244245
self._actual_key_value_store = actual_key_value_store
245246

246247
async def set_value(
@@ -250,7 +251,16 @@ async def set_value(
250251
*,
251252
content_type: str | None = None,
252253
) -> None:
253-
self.updates[key] = KeyValueStoreValue(value, content_type)
254+
if value is None:
255+
self.updates[key] = None
256+
return
257+
258+
# Import here to avoid circular imports.
259+
from crawlee.storages._utils import serialize_value # noqa: PLC0415
260+
261+
# Serialized right away, so that a value that can't be stored fails in the request handler.
262+
content, resolved_content_type = await serialize_value(value, content_type)
263+
self.updates[key] = KeyValueStoreValue(content, resolved_content_type)
254264

255265
@overload
256266
async def get_value(self, key: str) -> Any: ...
@@ -263,7 +273,23 @@ async def get_value(self, key: str, default_value: T | None = None) -> T | None:
263273

264274
async def get_value(self, key: str, default_value: T | None = None) -> T | None:
265275
if key in self.updates:
266-
return cast('T', self.updates[key].content)
276+
update = self.updates[key]
277+
278+
# A pending `None` deletes the record once the changes are committed.
279+
if update is None:
280+
return default_value
281+
282+
# Import here to avoid circular imports.
283+
from crawlee.storages._utils import parse_value # noqa: PLC0415
284+
285+
try:
286+
return cast('T', parse_value(update.content, update.content_type))
287+
except ValueError as exc:
288+
exc.add_note(
289+
f"The value set under key '{key}' in this request handler can't be parsed as "
290+
f"'{update.content_type}'."
291+
)
292+
raise
267293

268294
return await self._actual_key_value_store.get_value(key, default_value)
269295

‎src/crawlee/_utils/file.py‎

Lines changed: 18 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
import csv
55
import json
66
import os
7+
import re
78
import sys
89
import tempfile
910
from logging import getLogger
@@ -103,20 +104,27 @@ def infer_mime_type(value: Any) -> str:
103104
Returns:
104105
The inferred MIME content type.
105106
"""
106-
# If the value is bytes (or bytearray), return binary content type.
107-
if isinstance(value, (bytes, bytearray)):
107+
if isinstance(value, (bytes, bytearray, memoryview)):
108108
return 'application/octet-stream'
109109

110-
# If the value is a dict or list, assume JSON.
111-
if isinstance(value, (dict, list)):
112-
return 'application/json; charset=utf-8'
113-
114-
# If the value is a string, number or boolean, assume plain text.
115-
if isinstance(value, (str, int, float, bool)):
110+
if isinstance(value, str):
116111
return 'text/plain; charset=utf-8'
117112

118-
# Default fallback.
119-
return 'application/octet-stream'
113+
# Any other value is stored as JSON.
114+
return 'application/json; charset=utf-8'
115+
116+
117+
_CHARSET_PATTERN = re.compile(r'(?:^|;)\s*charset\s*=\s*(?:"([^"]*)"|([^;\s]*))', re.IGNORECASE)
118+
119+
120+
def parse_content_type_charset(value: str | None) -> str | None:
121+
"""Get the `charset` parameter of a MIME content type, such as a `Content-Type` header value, if it has one."""
122+
if not value:
123+
return None
124+
match = _CHARSET_PATTERN.search(value)
125+
if match is None:
126+
return None
127+
return (match.group(1) or match.group(2) or '').strip() or None
120128

121129

122130
async def json_dumps(obj: Any) -> str:

‎src/crawlee/_utils/html.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@
1212
from crawlee._request import Request
1313
from crawlee._types import HttpHeaders
1414
from crawlee._utils.crypto import compute_short_hash
15-
from crawlee._utils.http import parse_content_type_charset
15+
from crawlee._utils.file import parse_content_type_charset
1616
from crawlee._utils.urls import convert_to_absolute_url, is_url_absolute, validate_http_url
1717

1818
if TYPE_CHECKING:

‎src/crawlee/_utils/http.py‎

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,6 @@
33
from __future__ import annotations
44

55
import asyncio
6-
import re
76
from contextlib import suppress
87
from datetime import UTC, datetime, timedelta
98
from email.utils import parsedate_to_datetime
@@ -21,18 +20,6 @@
2120
_MAX_DRAIN_SECONDS = 1
2221
"""How long `drain_small_body` waits for the body."""
2322

24-
_CHARSET_PATTERN = re.compile(r'(?:^|;)\s*charset\s*=\s*(?:"([^"]*)"|([^;\s]*))', re.IGNORECASE)
25-
26-
27-
def parse_content_type_charset(value: str | None) -> str | None:
28-
"""Get the `charset` parameter of a `Content-Type` header value, if it has one."""
29-
if not value:
30-
return None
31-
match = _CHARSET_PATTERN.search(value)
32-
if match is None:
33-
return None
34-
return (match.group(1) or match.group(2) or '').strip() or None
35-
3623

3724
def parse_retry_after_header(value: str | None) -> timedelta | None:
3825
"""Parse the Retry-After HTTP header value.

‎src/crawlee/_utils/recoverable_state.py‎

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
from datetime import timedelta
55
from typing import TYPE_CHECKING, Generic, TypeVar
66

7-
from pydantic import BaseModel, ValidationError
7+
from pydantic import BaseModel
88

99
from crawlee._utils.raise_if_too_many_kwargs import raise_if_too_many_kwargs
1010
from crawlee._utils.wait import wait_for
@@ -112,8 +112,8 @@ async def initialize(self) -> TStateModel:
112112
The loaded state model
113113
114114
Raises:
115-
ValidationError: If the persisted record is invalid. The record is kept, and every call raises again until
116-
it's fixed or cleared by `reset_store`.
115+
ValueError: If the persisted record is invalid. The record is kept, and every call raises again until it's
116+
fixed or cleared by `reset_store`.
117117
"""
118118
# Checked without the lock first, so that a call on an initialized state doesn't wait for a save in progress.
119119
if self._initialized:
@@ -254,23 +254,22 @@ async def _open_key_value_store(self) -> KeyValueStore:
254254
return self._key_value_store
255255

256256
async def _load_saved_state(self) -> None:
257-
stored_state = None
258257
if self._persistence_enabled:
259258
key_value_store = await self._open_key_value_store()
260-
stored_state = await self._with_timeout(
261-
lambda: key_value_store.get_value(self._persist_state_key), 'Loading the persisted state'
262-
)
263-
264-
if stored_state is not None:
265259
try:
266-
self._state = self._state_type.model_validate(stored_state)
267-
except ValidationError as exc:
260+
stored_state = await self._with_timeout(
261+
lambda: key_value_store.get_value(self._persist_state_key), 'Loading the persisted state'
262+
)
263+
if stored_state is not None:
264+
self._state = self._state_type.model_validate(stored_state)
265+
except ValueError as exc:
268266
exc.add_note(
269267
f"The state persisted under key '{self._persist_state_key}' is invalid. Fix or delete the record "
270268
'to start from the default state.'
271269
)
272270
raise
273-
elif self._state is None:
271+
272+
if self._state is None:
274273
self._state = self._default_state.model_copy(deep=True)
275274

276275
# Set only once the load succeeds. A load that fails, is cancelled or finds an invalid record leaves the object

0 commit comments

Comments
 (0)