You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2c2d1f1
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/03_concepts/03_crawling_context.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -56,7 +56,7 @@ The following example uses most of these helpers in a <ApiLink to="class/ParselC
56
56
57
57
## Results are committed after the handler succeeds
58
58
59
-
The `push_data`, `add_requests` and `enqueue_links` calls and the `set_value` calls on a key-value store from `get_key_value_store` don't write anything right away. The context records them in an internal `RequestHandlerRunResult` object, and the crawler commits the recorded changes only after the request handler returns without an exception. Reading a key from that key-value store in the same handler returns the value you've set, even before the commit.
59
+
The `push_data`, `add_requests` and `enqueue_links` calls and the `set_value` calls on a key-value store from `get_key_value_store` don't write anything right away. The context records them in an internal `RequestHandlerRunResult` object, and the crawler commits the recorded changes only after the request handler returns without an exception. The key-value store keeps each value as it was when you called `set_value`, so changing the object afterwards doesn't change what's committed. Reading a key from that key-value store in the same handler returns the value you've set, even before the commit. It comes back the way the store reads it after the commit, so a tuple is read as a list.
60
60
61
61
When the handler raises, the crawler discards the recorded changes and retries the request. A handler that pushes an item and then fails on the next line doesn't leave a partial or duplicate item in the dataset. The retry starts with an empty record and produces the output once, when it succeeds. The same applies to requests added by a failed attempt.
Copy file name to clipboardExpand all lines: docs/03_concepts/26_extending_crawlee.mdx
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -148,7 +148,7 @@ A storage backend is the layer behind Crawlee's three storages. <ApiLink to="cla
148
148
The <ApiLinkto="class/StorageBackend">`StorageBackend`</ApiLink> contract defines three factory methods: `create_dataset_backend`, `create_kvs_backend`, and `create_rq_backend`. The returned backends define the rest of the contract:
149
149
150
150
- <ApiLinkto="class/DatasetBackend">`DatasetBackend`</ApiLink> handles appending and reading items.
151
-
- <ApiLinkto="class/KeyValueStoreBackend">`KeyValueStoreBackend`</ApiLink> handles record access and iteration.
151
+
- <ApiLinkto="class/KeyValueStoreBackend">`KeyValueStoreBackend`</ApiLink> handles reading and writing records as bytes and listing their keys a page at a time.
152
152
- <ApiLinkto="class/RequestQueueBackend">`RequestQueueBackend`</ApiLink> handles adding, fetching, and marking requests as handled.
Copy file name to clipboardExpand all lines: docs/06_upgrading/upgrading_to_v2.md
+59-2Lines changed: 59 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -176,9 +176,64 @@ Data written by v1 still loads, so an existing storage directory, Redis instance
176
176
177
177
## `MemoryStorageBackend` stores copies
178
178
179
-
`MemoryStorageBackend` stores a JSON snapshot of each dataset item and of each key-value store value with a JSON content type, serialized the same way as by `FileSystemStorageBackend`. Changing an object after you push or set it doesn't change the stored data, and pushing the same object twice stores two independent items. Reads return fresh copies too, so changing a returned item or value doesn't change the store.
179
+
`MemoryStorageBackend` stores a JSON snapshot of each dataset item, serialized the same way as by `FileSystemStorageBackend`. Changing an object after you push it doesn't change the stored data, and pushing the same object twice stores two independent items. Reads return fresh copies too, so changing a returned item doesn't change the dataset.
180
180
181
-
The memory backend now handles values the same way as the file-system backend. It raises on a value the file-system backend can't serialize, such as one with a circular reference or a tuple as a dict key. Values that JSON doesn't support, such as a `datetime`, are stored as strings, and dict keys that aren't strings become strings.
181
+
The memory backend now handles dataset items the same way as the file-system backend. It raises on an item the file-system backend can't serialize, such as one with a circular reference or a tuple as a dict key. Values that JSON doesn't support, such as a `datetime`, are stored as strings, and dict keys that aren't strings become strings.
182
+
183
+
Key-value store values are serialized by `KeyValueStore` itself, the same way for every storage backend. For details, see [`KeyValueStore` serializes and parses the values](#keyvaluestore-serializes-and-parses-the-values).
184
+
185
+
## `KeyValueStore` serializes and parses the values
186
+
187
+
`KeyValueStore` now serializes a value before passing it to the storage backend, and parses the stored bytes after reading them. In v1, each storage backend did both on its own. The rules are the same for every storage backend. The v1 behavior described below is that of the file-system, SQL and Redis backends. `MemoryStorageBackend` kept each value as it was set.
188
+
189
+
### `set_value` with `None` deletes the record
190
+
191
+
`set_value(key, None)` deletes the record, the same as `delete_value(key)`. In v1, it stored a record with the `application/x-none` content type, so `record_exists` returned `True` for it. A record stored that way by v1 is now read as empty `bytes`. Such records include the persisted state of `Statistics` and `SessionPool`, which v1 cleared this way when `run()` was called again after a finished run. If the process stopped before that state was saved again, the next start fails with a `ValidationError` for the record. Delete the record to start from the default state.
192
+
193
+
### Values without a content type
194
+
195
+
Without `content_type`, `bytes`, `bytearray` and `memoryview` values are stored as `application/octet-stream`, `str` values as `text/plain`, and any other value as JSON. In v1, only `dict` and `list` values were stored as JSON. Numbers and booleans were stored as text and read back as `str`, and other objects were stored as the bytes of their `str()` form.
196
+
197
+
```python
198
+
await kvs.set_value('count', 5)
199
+
200
+
# v1
201
+
await kvs.get_value('count') # '5'
202
+
203
+
# v2
204
+
await kvs.get_value('count') # 5
205
+
```
206
+
207
+
### Values with a content type
208
+
209
+
A `str` or bytes value is stored as it is under the given `content_type`. In v1, a value with a JSON content type was always serialized, so a string that already held JSON was encoded twice and read back as a string. In v2, it's read back parsed. A string that isn't valid JSON is stored too, and reading it raises `ValueError`. Store such a string without the JSON content type. `Dataset.export_to` with the JSON content type now writes the items as a JSON array, where v1 wrote that array wrapped in a JSON string.
210
+
211
+
A `str` value is encoded with the charset of its content type. A charset that Python doesn't know raises `LookupError`, and a character that the charset can't encode raises `UnicodeEncodeError`. In v1, a `str` value was always encoded as UTF-8. A `str` value that v1 stored under a content type with another charset is now decoded with that charset, so its non-ASCII characters come back wrong. To read its text, decode the bytes from the new `KeyValueStore.get_record` as UTF-8.
212
+
213
+
Any other value can only be stored as JSON. Passing it with a content type that isn't JSON raises `TypeError`. In v1, such a value was stored as the bytes of its `str()` form.
214
+
215
+
### Reading values
216
+
217
+
`get_value` parses a record by its content type. `application/json` is parsed, `text/*` and `application/*xml` are decoded to `str` with the charset of the content type, or with UTF-8 when it has none, and any other content type is returned as `bytes`. A JSON or text record with a charset that Python doesn't know is returned as `bytes` too. In v1, `application/*xml` content types were returned as `bytes`, and the charset was ignored.
218
+
219
+
A record with a JSON content type that isn't valid JSON raises `ValueError` from `get_value`, `iterate_values` and `iterate_entries`. In a text or JSON record, the bytes that its charset can't decode are replaced with the Unicode replacement character. In v1, the storage backend logged a warning in both cases, and the record was reported as missing. To read the stored bytes of such a record, use `KeyValueStore.get_record`.
220
+
221
+
### Values set in a request handler
222
+
223
+
The key-value store from `context.get_key_value_store()` serializes a value when you call `set_value`. Changes made to the object afterwards aren't stored when the crawler commits the results of the handler, and `get_value` in the same handler returns a new copy of the value. In v1, the store kept the object itself until the commit, so those changes were stored, and `get_value` returned the same object. A value that can't be stored now raises from `set_value` in the handler, where v1 raised only during the commit.
224
+
225
+
### Custom key-value store backends
226
+
227
+
A custom `KeyValueStoreBackend` no longer serializes or parses the values:
228
+
229
+
-`set_value(*, key, value, content_type)` receives the value as `bytes`, and `content_type` is always set.
230
+
-`get_value(*, key)` returns a `KeyValueStoreRecord` whose `value` is the stored `bytes`. `KeyValueStoreRecord` is no longer generic.
231
+
-`list_keys(*, prefix, exclusive_start_key, limit)` replaces `iterate_keys` and `iterate_entries`. It returns a single `KeyValueStoreKeysListPage`, whose `items` are ordered by key. Its `next_exclusive_start_key` is where the next page starts, or `None` on the last page. A page before the last one may hold fewer than `limit` keys, or none. The page also carries `count`, the number of keys on it, and the `limit` and `exclusive_start_key` it was requested with.
232
+
-`get_values(*, keys)` is optional. Override it if the backend can read several records in a single call. It returns a mapping from each key that has a record to that record. The default implementation calls `get_value` for each key. The built-in file-system, SQL and Redis backends override it, so a subclass of one of them that changes `get_value` must change `get_values` the same way.
233
+
234
+
### `RedisStorageBackend` indexes the record keys
235
+
236
+
`RedisStorageBackend` keeps the keys of each key-value store in a sorted set, `key_value_stores:{name}:keys`, so it lists them a page at a time. When v2 opens a store written by v1, it builds the set from the existing records. Don't write to the same store from v1 and v2 at the same time, since v2 doesn't list the keys that v1 adds after the set exists.
182
237
183
238
## SQL and Redis lock fetched requests for 3 minutes
184
239
@@ -688,6 +743,8 @@ Messages that mentioned a storage client name the storage backend instead. The D
688
743
689
744
`RequestQueue.purge()` no longer logs the `Storage client "..." does not support purging the request queue.` warnings. It raises `NotImplementedError` instead. For details, see [`RequestQueue.purge` raises when the storage backend can't purge](#requestqueuepurge-raises-when-the-storage-backend-cant-purge).
690
745
746
+
The storage backends no longer log the `Failed to decode JSON value for key "..."` and `Failed to decode text value for key "..."` warnings. Instead, `KeyValueStore` raises `ValueError` for a record that isn't valid JSON, and replaces the bytes it can't decode in a text or JSON record. For details, see [Reading values](#reading-values). `RedisStorageBackend` no longer logs the `Metadata for key "..." is missing or invalid.` warning when it reads a key that has no record.
747
+
691
748
The ERROR message the crawler logs before raising `PersistentRateLimitError` now reads "Giving up the crawl because the request manager can't make progress." instead of "Giving up the crawl because a domain keeps rate-limiting every request."
692
749
693
750
The two DEBUG messages that `RequestQueue.is_finished` logged, "Background add requests tasks are still in progress." and "The request queue is finished.", are removed together with the method.
0 commit comments