Bug: `_normalise_search_criteria()` does not propagate `charset` in recursive calls

[uppersafe]

Description

When using IMAPClient.search() with nested search criteria and a non-ASCII charset such as UTF-8, _normalise_search_criteria() does not propagate the charset argument when recursively processing nested lists or tuples.

As a result, nested criteria fall back to the default "us-ascii" charset instead of using the charset provided by the caller.

The relevant code currently looks like this:

elif isinstance(item, (list, tuple)):
    # Process nested criteria list and wrap in parens.
    inner = _normalise_search_criteria(item)

I believe the recursive call should pass the current charset:

elif isinstance(item, (list, tuple)):
    # Process nested criteria list and wrap in parens.
    inner = _normalise_search_criteria(item, charset)

Example

For example:

criteria = [
    "OR",
    ["TEXT", "test"],
    [
        "OR",
        ["TEXT", "signé"],
        ["TEXT", "école"],
    ],
]

client.search(criteria, charset="UTF-8")

The top-level call correctly uses "UTF-8", but when _normalise_search_criteria() processes a nested list, it calls itself without passing charset.

Since the function defaults to "us-ascii" when no charset is provided, nested search criteria are therefore normalized using the wrong charset.

Current behavior

A recursive call such as:

_normalise_search_criteria(item)

causes the nested invocation to fall back to:

charset = "us-ascii"

This can lead to incorrect handling of non-ASCII search terms such as:

signé
école

Resulting in the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<string>", line 351, in _browse_imap
> File "<string>", line 127, in search
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1122, in search
    return self._search(criteria, charset)
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1145, in _search
    args.extend(_normalise_search_criteria(criteria, charset))
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1847, in _normalise_search_criteria
    inner = _normalise_search_criteria(item)
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1847, in _normalise_search_criteria
    inner = _normalise_search_criteria(item)
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1852, in _normalise_search_criteria
    out.append(_quoted.maybe(to_bytes(item, charset)))
  File "/usr/local/lib/python3.11/site-packages/imapclient/util.py", line 29, in to_bytes
    return s.encode(charset)
UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' in position 4: ordinal not in range(128)

Also, even by encoding the criteria before passing it to the search function, I encountered another exception:

criteria = [
    "OR",
    ["TEXT", "test"],
    [
        "OR",
        ["TEXT", "signé".encode("utf-8")],
        ["TEXT", "école".encode("utf-8")],
    ],
]

client.search(criteria, charset="UTF-8")

Resulting in an invalid SEARCH / UID SEARCH command being sent to the IMAP server.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/threading.py", line 1002, in _bootstrap
    self._bootstrap_inner()
  File "/usr/local/lib/python3.11/threading.py", line 1045, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.11/threading.py", line 982, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 83, in _worker
    work_item.run()
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "<string>", line 352, in _browse_imap
> File "<string>", line 127, in search
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1122, in search
    return self._search(criteria, charset)
  File "/usr/local/lib/python3.11/site-packages/imapclient/imapclient.py", line 1153, in _search
    raise exceptions.InvalidCriteriaError(
imapclient.exceptions.InvalidCriteriaError: b"Error in IMAP command UID SEARCH: Missing ')' (0.001 + 0.000 secs)."

This error may have been caused by a syntax error in the criteria: ['OR', ['TEXT', 'test'], ['OR', ['TEXT', b'sign\xc3\xa9'], ['TEXT', b'\xc3\xa9cole']]]
Please refer to the documentation for more information about search criteria syntax..
https://imapclient.readthedocs.io/en/master/#imapclient.IMAPClient.search

Expected behavior

The charset passed to search() should be preserved while recursively normalizing all nested search criteria.

Suggested fix

elif isinstance(item, (list, tuple)):
    # Process nested criteria list and wrap in parens.
-   inner = _normalise_search_criteria(item)
+   inner = _normalise_search_criteria(item, charset)

A regression test using nested TEXT criteria containing UTF-8 characters would probably also be useful.

Environment

• Python: 3.11