python_utils-4.0.0/PKG-INFO0000644000000350760000000000000107530ustar Metadata-Version: 2.4 Name: python-utils Version: 4.0.0 Summary: Python Utils is a module with some convenient utilities not included with the standard Python install Keywords: utils,utilities,helpers,typing,typed,async,lazy-imports Author: Rick van Hattem Author-email: Rick van Hattem License-Expression: BSD-3-Clause License-File: LICENSE Classifier: Development Status :: 5 - Production/Stable Classifier: Intended Audience :: Developers Classifier: Operating System :: OS Independent Classifier: Programming Language :: Python :: 3 Classifier: Programming Language :: Python :: 3.10 Classifier: Programming Language :: Python :: 3.11 Classifier: Programming Language :: Python :: 3.12 Classifier: Programming Language :: Python :: 3.13 Classifier: Programming Language :: Python :: 3.14 Classifier: Programming Language :: Python :: 3 :: Only Classifier: Programming Language :: Python :: Implementation :: CPython Classifier: Programming Language :: Python :: Implementation :: PyPy Classifier: Typing :: Typed Requires-Dist: typing-extensions>3.10.0.2 Requires-Dist: sphinx ; extra == 'docs' Requires-Dist: furo ; extra == 'docs' Requires-Dist: myst-parser ; extra == 'docs' Requires-Dist: loguru ; extra == 'loguru' Requires-Dist: pytest ; extra == 'tests' Requires-Dist: pytest-cov ; extra == 'tests' Requires-Dist: pytest-asyncio ; extra == 'tests' Requires-Dist: loguru ; extra == 'tests' Requires-Dist: blessings ; extra == 'tests' Requires-Python: >=3.10 Project-URL: Homepage, https://github.com/WoLpH/python-utils Project-URL: Documentation, https://python-utils.readthedocs.io/ Project-URL: Repository, https://github.com/WoLpH/python-utils Project-URL: Changelog, https://github.com/WoLpH/python-utils/releases Provides-Extra: docs Provides-Extra: loguru Provides-Extra: tests Description-Content-Type: text/markdown
python-utils # ⚡ Python Utils **The fast, fully-typed stdlib helpers you keep rewriting — in one tiny, dependency-light package.** [![PyPI version](https://img.shields.io/pypi/v/python-utils.svg?logo=pypi&logoColor=white)](https://pypi.python.org/pypi/python-utils) [![Python versions](https://img.shields.io/pypi/pyversions/python-utils.svg?logo=python&logoColor=white)](https://pypi.python.org/pypi/python-utils) [![CI](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml) [![Coverage Status](https://coveralls.io/repos/github/WoLpH/python-utils/badge.svg?branch=develop)](https://coveralls.io/github/WoLpH/python-utils?branch=develop) [![Typed](https://img.shields.io/badge/typed-mypy%20%7C%20pyright%20%7C%20pyrefly-blue.svg)](https://github.com/WoLpH/python-utils) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) [![License](https://img.shields.io/pypi/l/python-utils.svg)](https://github.com/WoLpH/python-utils/blob/develop/LICENSE) [![Downloads](https://img.shields.io/pypi/dm/python-utils.svg?logo=pypi&logoColor=white)](https://pypi.python.org/pypi/python-utils) [**Documentation**](https://python-utils.readthedocs.io/en/latest/) · [**PyPI**](https://pypi.python.org/pypi/python-utils) · [**Source**](https://github.com/WoLpH/python-utils) · [**Issues**](https://github.com/WoLpH/python-utils/issues)
--- Python Utils is a collection of small, battle-tested functions and classes that make everyday Python patterns shorter, safer and faster. No sprawling framework, no heavy dependencies — just the helpers you find yourself re-writing in project after project, packaged once and typed to the hilt. It has powered production code for years (and is used by libraries such as [Django Utils](https://pypi.python.org/pypi/django-utils2) and [progressbar2](https://pypi.python.org/pypi/progressbar2)). ## ✨ Highlights - 🪶 **Zero-cost imports** — thanks to [PEP 562][pep562] lazy loading, `import python_utils` pulls in *nothing* until you actually touch a helper. No `asyncio`, no `typing_extensions`, until you ask for them. - ⚡ **Async-native** — `acount`, `abatcher`, and timeout/stall detectors bring `itertools`-style ergonomics to `async for`. - 📦 **Smart containers** — self-casting dicts, duplicate-proof lists and a sliceable deque. - 🔢 **Forgiving converters** — pull an `int`/`float` out of *any* messy string, scale bytes to KiB/MiB, remap values between ranges (with `Decimal` precision). - ⏱️ **Time & retries** — human-readable durations plus timeout generators for sampling slow APIs without hanging. - 🎯 **Fully typed & 100% covered** — ships `py.typed`, passes **mypy**, **basedpyright** *and* **pyrefly** in strict mode, with 100% test coverage. - 🐍 **Modern & tiny** — Python 3.10+, a single runtime dependency (`typing_extensions`), BSD-3 licensed. ## 🗺️ What's inside | Module | What you get | | --- | --- | | [`converters`](https://python-utils.readthedocs.io/en/latest/) | `to_int` · `to_float` · `to_str` · `to_unicode` · `scale_1024` · `remap` | | [`formatters`](https://python-utils.readthedocs.io/en/latest/) | `camel_to_underscore` · `apply_recursive` · `timesince` | | [`time`](https://python-utils.readthedocs.io/en/latest/) | `format_time` · `timeout_generator` · `aio_timeout_generator` · `aio_generator_timeout_detector` | | [`generators`](https://python-utils.readthedocs.io/en/latest/) | `batcher` · `abatcher` (batch by size **or** time interval) | | [`aio`](https://python-utils.readthedocs.io/en/latest/) | `acount` · `acontainer` — async `itertools` | | [`containers`](https://python-utils.readthedocs.io/en/latest/) | `CastedDict` · `LazyCastedDict` · `UniqueList` · `SliceableDeque` | | [`decorators`](https://python-utils.readthedocs.io/en/latest/) | `listify` · `set_attributes` · `sample` · `wraps_classmethod` | | [`logger`](https://python-utils.readthedocs.io/en/latest/) | `Logged` · `LoggerBase` (+ `Logurud` via the `loguru` extra) | | [`import_`](https://python-utils.readthedocs.io/en/latest/) | `import_global` — programmatic `from x import *` | | [`exceptions`](https://python-utils.readthedocs.io/en/latest/) | `raise_exception` · `reraise` | | [`terminal`](https://python-utils.readthedocs.io/en/latest/) | `get_terminal_size` — works in shells, IPython & Jupyter | | [`types`](https://python-utils.readthedocs.io/en/latest/) | handy type aliases (`Number`, `Scope`, `StringTypes`, …) | ## 📦 Installation ```bash pip install python-utils ``` Optional extras: ```bash pip install 'python-utils[loguru]' # loguru-backed logging mixin ``` Python **3.10+** is required. The only runtime dependency is `typing_extensions` (and it's imported lazily). ## 🚀 Quickstart ```python import python_utils # Pull a number out of any messy string python_utils.to_int('listening on port=8080', regexp=True) # 8080 # Human-readable sizes: (value, power-of-1024) python_utils.scale_1024(1536, 2) # (1.5, 1) -> 1.5 KiB # Remap a value between ranges (46% volume -> dB on an AVR) python_utils.remap(46.0, 0.0, 100.0, -80.0, 10.0) # -38.6 # "time ago" formatting, Django-style import datetime python_utils.timesince(datetime.datetime.now() - datetime.timedelta(seconds=61)) # '1 minute and 1 second ago' ``` Everything is reachable straight off the top-level package (`python_utils.`) or from its submodule (`python_utils.converters.to_int`) — pick whichever reads better. Either way, only the modules you touch get imported. ## 🧰 Examples
🔢 Converters — numbers out of anything ```python from python_utils import converters # Extract digits with a built-in or custom regexp converters.to_int('spam15eggs', regexp=True) # 15 converters.to_int('nope', default=-1) # -1 converters.to_float('pi is 3.14', regexp=True) # 3.14 # Scale bytes to a sensible unit (value, power) -> 2.0 KiB converters.scale_1024(2048, 3) # (2.0, 1) # Linear remap; pass a Decimal anywhere to keep full precision converters.remap(500, 0, 1000, 0, 100) # 50 import decimal converters.remap(decimal.Decimal('250.0'), 0.0, 1000.0, 0.0, 100.0) # Decimal('25.0') ```
📦 Containers — dicts & lists with super-powers ```python from python_utils import containers # Keys and values are cast on the way in d = containers.CastedDict(int, int) d['3'] = '4' d.update({'5': '6'}) d # {3: 4, 5: 6} # A list that silently drops duplicates (or raises, if you prefer) u = containers.UniqueList(1, 2, 3) u.append(2) # ignored u # [1, 2, 3] # A deque you can actually slice s = containers.SliceableDeque([1, 2, 3, 4, 5]) s[1:4] # SliceableDeque([2, 3, 4]) ```
⚡ Async helpers — itertools for async for ```python from python_utils import aio, generators # Async counter (optionally with a delay and a stop value) async def demo(): async for i in aio.acount(stop=3): print(i) # 0, 1, 2 # Batch an async stream by size OR time interval — whichever comes first. # Great for chunking bursty producers without ever stalling a slow loop. async def batched(): async for batch in generators.abatcher(aio.acount(stop=10), batch_size=3): print(batch) # [0, 1, 2], [3, 4, 5], [6, 7, 8], [9] # Sync batching too: list(generators.batcher(range(9), 3)) # [[0, 1, 2], [3, 4, 5], [6, 7, 8]] ```
⏱️ Time & retries — sample slow APIs, format durations ```python import datetime from python_utils import time # Loop over a slow operation, but give up after `timeout` seconds for i in time.timeout_generator(0.1, interval=0.06): ... # yields 0, 1, 2 then stops # Format timedeltas, datetimes and raw seconds uniformly time.format_time(1) # '0:00:01' time.format_time(datetime.timedelta(seconds=3661)) # '1:01:01' time.format_time(datetime.datetime(2000, 1, 2, 3, 4, 5)) # '2000-01-02 03:04:05' time.format_time(None) # '--:--:--' ``` There's also `aio_timeout_generator` (the `async for` twin) and `aio_generator_timeout_detector`, which fails fast when an async generator stalls instead of hanging forever.
🔤 Formatters — case conversion & friendly timestamps ```python from python_utils import formatters formatters.camel_to_underscore('SpamEggsAndBacon') # 'spam_eggs_and_bacon' # Recursively rewrite every key in a nested dict formatters.apply_recursive( formatters.camel_to_underscore, {'SpamEggs': {'FooBar': 1}}, ) # {'spam_eggs': {'foo_bar': 1}} ```
🎀 Decorators — collect generators, tag functions, sample calls ```python from python_utils import decorators # Turn a generator into a concrete collection automatically @decorators.listify() def numbers(): yield 1 yield 2 yield 3 numbers() # [1, 2, 3] @decorators.listify(collection=dict) def pairs(): yield 'a', 1 yield 'b', 2 pairs() # {'a': 1, 'b': 2} # Attach metadata to a function (handy for the Django admin) @decorators.set_attributes(short_description='Name') def upper_case_name(self, obj): return f'{obj.first_name} {obj.last_name}'.upper() # Only actually run ~10% of the calls @decorators.sample(0.1) def maybe_log(msg): ... ```
📝 Logging — a correctly-named logger on every class ```python from python_utils.logger import Logged class MyClass(Logged): def do_work(self): self.info('starting %s', 'work') # stdlib %-style logging args self.error('something went wrong') MyClass().do_work() ``` Prefer [loguru](https://github.com/Delgan/loguru)? Install the extra (`pip install 'python-utils[loguru]'`) and subclass `Logurud` instead — the same `self.info(...)` / `self.error(...)` API, backed by loguru so you keep all its configuration and per-instance context.
🖥️ Terminal & 🧩 misc ```python from python_utils import terminal, import_ from python_utils.exceptions import raise_exception, reraise # Robust terminal size (tries IPython/Jupyter, shutil, blessings, ioctl, tput…) terminal.get_terminal_size() # e.g. (80, 24) # Programmatic `from some_module import *` import_.import_global('os') # Build a callable that raises — useful as a default/callback on_error = raise_exception(ValueError, 'boom') ```
## ⚡ Performance: lazy by default `import python_utils` is intentionally *cheap*. Every submodule and every export is wired through a [PEP 562][pep562] `__getattr__`, so nothing is imported until first access — and then it's cached. In particular: - Need only the synchronous helpers? `asyncio` is never imported. - Even `typing_extensions` is deferred, so the import graph stays minimal. ```python import sys import python_utils # imports basically nothing extra 'asyncio' in sys.modules # False python_utils.acount # now `aio` (and asyncio) load, on demand ``` See the [performance guide](https://python-utils.readthedocs.io/en/latest/) for the full story. ## 📚 Documentation Full API reference and guides live at ****. ## 🔗 Links - 📖 Documentation: - 🐙 Source: - 📦 PyPI: - 🐛 Issues: - ✍️ Author's blog: ## 🔒 Security To report a security vulnerability, please use the [Tidelift security contact](https://tidelift.com/security). Tidelift will coordinate the fix and disclosure. ## 🤝 Contributing Contributions are very welcome! We keep a strict **100% coverage** bar and run `ruff`, three type checkers and the full test matrix in CI. See [CONTRIBUTING.md](https://github.com/WoLpH/python-utils/blob/develop/CONTRIBUTING.md) to get set up. ## 📄 License BSD-3-Clause — see [LICENSE](https://github.com/WoLpH/python-utils/blob/develop/LICENSE). [pep562]: https://peps.python.org/pep-0562/ python_utils-4.0.0/0000755000000000000000000000000076435ustar python_utils-4.0.0/LICENSE0000644000000027350000000000000106570ustar Copyright (c) 2016, Rick van Hattem All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: 1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. python_utils-4.0.0/README.md0000644000000314040000000000000111240ustar
python-utils # ⚡ Python Utils **The fast, fully-typed stdlib helpers you keep rewriting — in one tiny, dependency-light package.** [![PyPI version](https://img.shields.io/pypi/v/python-utils.svg?logo=pypi&logoColor=white)](https://pypi.python.org/pypi/python-utils) [![Python versions](https://img.shields.io/pypi/pyversions/python-utils.svg?logo=python&logoColor=white)](https://pypi.python.org/pypi/python-utils) [![CI](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml/badge.svg)](https://github.com/WoLpH/python-utils/actions/workflows/ci.yml) [![Coverage Status](https://coveralls.io/repos/github/WoLpH/python-utils/badge.svg?branch=develop)](https://coveralls.io/github/WoLpH/python-utils?branch=develop) [![Typed](https://img.shields.io/badge/typed-mypy%20%7C%20pyright%20%7C%20pyrefly-blue.svg)](https://github.com/WoLpH/python-utils) [![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) [![License](https://img.shields.io/pypi/l/python-utils.svg)](https://github.com/WoLpH/python-utils/blob/develop/LICENSE) [![Downloads](https://img.shields.io/pypi/dm/python-utils.svg?logo=pypi&logoColor=white)](https://pypi.python.org/pypi/python-utils) [**Documentation**](https://python-utils.readthedocs.io/en/latest/) · [**PyPI**](https://pypi.python.org/pypi/python-utils) · [**Source**](https://github.com/WoLpH/python-utils) · [**Issues**](https://github.com/WoLpH/python-utils/issues)
--- Python Utils is a collection of small, battle-tested functions and classes that make everyday Python patterns shorter, safer and faster. No sprawling framework, no heavy dependencies — just the helpers you find yourself re-writing in project after project, packaged once and typed to the hilt. It has powered production code for years (and is used by libraries such as [Django Utils](https://pypi.python.org/pypi/django-utils2) and [progressbar2](https://pypi.python.org/pypi/progressbar2)). ## ✨ Highlights - 🪶 **Zero-cost imports** — thanks to [PEP 562][pep562] lazy loading, `import python_utils` pulls in *nothing* until you actually touch a helper. No `asyncio`, no `typing_extensions`, until you ask for them. - ⚡ **Async-native** — `acount`, `abatcher`, and timeout/stall detectors bring `itertools`-style ergonomics to `async for`. - 📦 **Smart containers** — self-casting dicts, duplicate-proof lists and a sliceable deque. - 🔢 **Forgiving converters** — pull an `int`/`float` out of *any* messy string, scale bytes to KiB/MiB, remap values between ranges (with `Decimal` precision). - ⏱️ **Time & retries** — human-readable durations plus timeout generators for sampling slow APIs without hanging. - 🎯 **Fully typed & 100% covered** — ships `py.typed`, passes **mypy**, **basedpyright** *and* **pyrefly** in strict mode, with 100% test coverage. - 🐍 **Modern & tiny** — Python 3.10+, a single runtime dependency (`typing_extensions`), BSD-3 licensed. ## 🗺️ What's inside | Module | What you get | | --- | --- | | [`converters`](https://python-utils.readthedocs.io/en/latest/) | `to_int` · `to_float` · `to_str` · `to_unicode` · `scale_1024` · `remap` | | [`formatters`](https://python-utils.readthedocs.io/en/latest/) | `camel_to_underscore` · `apply_recursive` · `timesince` | | [`time`](https://python-utils.readthedocs.io/en/latest/) | `format_time` · `timeout_generator` · `aio_timeout_generator` · `aio_generator_timeout_detector` | | [`generators`](https://python-utils.readthedocs.io/en/latest/) | `batcher` · `abatcher` (batch by size **or** time interval) | | [`aio`](https://python-utils.readthedocs.io/en/latest/) | `acount` · `acontainer` — async `itertools` | | [`containers`](https://python-utils.readthedocs.io/en/latest/) | `CastedDict` · `LazyCastedDict` · `UniqueList` · `SliceableDeque` | | [`decorators`](https://python-utils.readthedocs.io/en/latest/) | `listify` · `set_attributes` · `sample` · `wraps_classmethod` | | [`logger`](https://python-utils.readthedocs.io/en/latest/) | `Logged` · `LoggerBase` (+ `Logurud` via the `loguru` extra) | | [`import_`](https://python-utils.readthedocs.io/en/latest/) | `import_global` — programmatic `from x import *` | | [`exceptions`](https://python-utils.readthedocs.io/en/latest/) | `raise_exception` · `reraise` | | [`terminal`](https://python-utils.readthedocs.io/en/latest/) | `get_terminal_size` — works in shells, IPython & Jupyter | | [`types`](https://python-utils.readthedocs.io/en/latest/) | handy type aliases (`Number`, `Scope`, `StringTypes`, …) | ## 📦 Installation ```bash pip install python-utils ``` Optional extras: ```bash pip install 'python-utils[loguru]' # loguru-backed logging mixin ``` Python **3.10+** is required. The only runtime dependency is `typing_extensions` (and it's imported lazily). ## 🚀 Quickstart ```python import python_utils # Pull a number out of any messy string python_utils.to_int('listening on port=8080', regexp=True) # 8080 # Human-readable sizes: (value, power-of-1024) python_utils.scale_1024(1536, 2) # (1.5, 1) -> 1.5 KiB # Remap a value between ranges (46% volume -> dB on an AVR) python_utils.remap(46.0, 0.0, 100.0, -80.0, 10.0) # -38.6 # "time ago" formatting, Django-style import datetime python_utils.timesince(datetime.datetime.now() - datetime.timedelta(seconds=61)) # '1 minute and 1 second ago' ``` Everything is reachable straight off the top-level package (`python_utils.`) or from its submodule (`python_utils.converters.to_int`) — pick whichever reads better. Either way, only the modules you touch get imported. ## 🧰 Examples
🔢 Converters — numbers out of anything ```python from python_utils import converters # Extract digits with a built-in or custom regexp converters.to_int('spam15eggs', regexp=True) # 15 converters.to_int('nope', default=-1) # -1 converters.to_float('pi is 3.14', regexp=True) # 3.14 # Scale bytes to a sensible unit (value, power) -> 2.0 KiB converters.scale_1024(2048, 3) # (2.0, 1) # Linear remap; pass a Decimal anywhere to keep full precision converters.remap(500, 0, 1000, 0, 100) # 50 import decimal converters.remap(decimal.Decimal('250.0'), 0.0, 1000.0, 0.0, 100.0) # Decimal('25.0') ```
📦 Containers — dicts & lists with super-powers ```python from python_utils import containers # Keys and values are cast on the way in d = containers.CastedDict(int, int) d['3'] = '4' d.update({'5': '6'}) d # {3: 4, 5: 6} # A list that silently drops duplicates (or raises, if you prefer) u = containers.UniqueList(1, 2, 3) u.append(2) # ignored u # [1, 2, 3] # A deque you can actually slice s = containers.SliceableDeque([1, 2, 3, 4, 5]) s[1:4] # SliceableDeque([2, 3, 4]) ```
⚡ Async helpers — itertools for async for ```python from python_utils import aio, generators # Async counter (optionally with a delay and a stop value) async def demo(): async for i in aio.acount(stop=3): print(i) # 0, 1, 2 # Batch an async stream by size OR time interval — whichever comes first. # Great for chunking bursty producers without ever stalling a slow loop. async def batched(): async for batch in generators.abatcher(aio.acount(stop=10), batch_size=3): print(batch) # [0, 1, 2], [3, 4, 5], [6, 7, 8], [9] # Sync batching too: list(generators.batcher(range(9), 3)) # [[0, 1, 2], [3, 4, 5], [6, 7, 8]] ```
⏱️ Time & retries — sample slow APIs, format durations ```python import datetime from python_utils import time # Loop over a slow operation, but give up after `timeout` seconds for i in time.timeout_generator(0.1, interval=0.06): ... # yields 0, 1, 2 then stops # Format timedeltas, datetimes and raw seconds uniformly time.format_time(1) # '0:00:01' time.format_time(datetime.timedelta(seconds=3661)) # '1:01:01' time.format_time(datetime.datetime(2000, 1, 2, 3, 4, 5)) # '2000-01-02 03:04:05' time.format_time(None) # '--:--:--' ``` There's also `aio_timeout_generator` (the `async for` twin) and `aio_generator_timeout_detector`, which fails fast when an async generator stalls instead of hanging forever.
🔤 Formatters — case conversion & friendly timestamps ```python from python_utils import formatters formatters.camel_to_underscore('SpamEggsAndBacon') # 'spam_eggs_and_bacon' # Recursively rewrite every key in a nested dict formatters.apply_recursive( formatters.camel_to_underscore, {'SpamEggs': {'FooBar': 1}}, ) # {'spam_eggs': {'foo_bar': 1}} ```
🎀 Decorators — collect generators, tag functions, sample calls ```python from python_utils import decorators # Turn a generator into a concrete collection automatically @decorators.listify() def numbers(): yield 1 yield 2 yield 3 numbers() # [1, 2, 3] @decorators.listify(collection=dict) def pairs(): yield 'a', 1 yield 'b', 2 pairs() # {'a': 1, 'b': 2} # Attach metadata to a function (handy for the Django admin) @decorators.set_attributes(short_description='Name') def upper_case_name(self, obj): return f'{obj.first_name} {obj.last_name}'.upper() # Only actually run ~10% of the calls @decorators.sample(0.1) def maybe_log(msg): ... ```
📝 Logging — a correctly-named logger on every class ```python from python_utils.logger import Logged class MyClass(Logged): def do_work(self): self.info('starting %s', 'work') # stdlib %-style logging args self.error('something went wrong') MyClass().do_work() ``` Prefer [loguru](https://github.com/Delgan/loguru)? Install the extra (`pip install 'python-utils[loguru]'`) and subclass `Logurud` instead — the same `self.info(...)` / `self.error(...)` API, backed by loguru so you keep all its configuration and per-instance context.
🖥️ Terminal & 🧩 misc ```python from python_utils import terminal, import_ from python_utils.exceptions import raise_exception, reraise # Robust terminal size (tries IPython/Jupyter, shutil, blessings, ioctl, tput…) terminal.get_terminal_size() # e.g. (80, 24) # Programmatic `from some_module import *` import_.import_global('os') # Build a callable that raises — useful as a default/callback on_error = raise_exception(ValueError, 'boom') ```
## ⚡ Performance: lazy by default `import python_utils` is intentionally *cheap*. Every submodule and every export is wired through a [PEP 562][pep562] `__getattr__`, so nothing is imported until first access — and then it's cached. In particular: - Need only the synchronous helpers? `asyncio` is never imported. - Even `typing_extensions` is deferred, so the import graph stays minimal. ```python import sys import python_utils # imports basically nothing extra 'asyncio' in sys.modules # False python_utils.acount # now `aio` (and asyncio) load, on demand ``` See the [performance guide](https://python-utils.readthedocs.io/en/latest/) for the full story. ## 📚 Documentation Full API reference and guides live at ****. ## 🔗 Links - 📖 Documentation: - 🐙 Source: - 📦 PyPI: - 🐛 Issues: - ✍️ Author's blog: ## 🔒 Security To report a security vulnerability, please use the [Tidelift security contact](https://tidelift.com/security). Tidelift will coordinate the fix and disclosure. ## 🤝 Contributing Contributions are very welcome! We keep a strict **100% coverage** bar and run `ruff`, three type checkers and the full test matrix in CI. See [CONTRIBUTING.md](https://github.com/WoLpH/python-utils/blob/develop/CONTRIBUTING.md) to get set up. ## 📄 License BSD-3-Clause — see [LICENSE](https://github.com/WoLpH/python-utils/blob/develop/LICENSE). [pep562]: https://peps.python.org/pep-0562/ python_utils-4.0.0/_python_utils_tests0000755000000000000000000000000137065ustar python_utils-4.0.0/_python_utils_tests/__init__.py0000644000000000430000000000000160730ustar """Test suite for python_utils.""" python_utils-4.0.0/_python_utils_tests/requirements.txt0000644000000000140000000000000172440ustar -e .[tests] python_utils-4.0.0/_python_utils_tests/test_aio.py0000644000000052040000000000000161470ustar """Tests for the async helpers in ``python_utils.aio``.""" import asyncio import pytest from python_utils import aio, types @pytest.mark.asyncio async def test_acount(monkeypatch: pytest.MonkeyPatch) -> None: """Count with a delay between yields until reaching ``stop``.""" sleeps: types.List[float] = [] async def mock_sleep(delay: float) -> None: """Record each requested delay instead of sleeping.""" sleeps.append(delay) monkeypatch.setattr(asyncio, 'sleep', mock_sleep) async for _i in aio.acount(delay=1, stop=3.5): pass assert len(sleeps) == 4 assert sum(sleeps) == 4 @pytest.mark.asyncio async def test_acontainer() -> None: """Collect an async iterable into the requested container.""" async def async_gen() -> types.AsyncIterable[int]: """Yield 1, 2, 3 asynchronously.""" yield 1 yield 2 yield 3 async def empty_gen() -> types.AsyncIterable[int]: """Yield nothing as an async generator.""" if False: yield 1 assert await aio.acontainer(async_gen) == [1, 2, 3] assert await aio.acontainer(async_gen()) == [1, 2, 3] assert await aio.acontainer(async_gen, set) == {1, 2, 3} assert await aio.acontainer(async_gen(), set) == {1, 2, 3} assert await aio.acontainer(async_gen, list) == [1, 2, 3] assert await aio.acontainer(async_gen(), list) == [1, 2, 3] assert await aio.acontainer(async_gen, tuple) == (1, 2, 3) assert await aio.acontainer(async_gen(), tuple) == (1, 2, 3) assert await aio.acontainer(empty_gen) == [] assert await aio.acontainer(empty_gen()) == [] assert await aio.acontainer(empty_gen, set) == set() assert await aio.acontainer(empty_gen(), set) == set() assert await aio.acontainer(empty_gen, list) == list() assert await aio.acontainer(empty_gen(), list) == list() assert await aio.acontainer(empty_gen, tuple) == tuple() assert await aio.acontainer(empty_gen(), tuple) == tuple() @pytest.mark.asyncio async def test_adict() -> None: """Build a dict from an async iterable of key/value pairs.""" async def async_gen() -> types.AsyncIterable[types.Tuple[int, int]]: """Yield key/value pairs asynchronously.""" yield 1, 2 yield 3, 4 yield 5, 6 async def empty_gen() -> types.AsyncIterable[types.Tuple[int, int]]: """Yield no pairs as an async generator.""" if False: yield 1, 2 assert await aio.adict(async_gen) == {1: 2, 3: 4, 5: 6} assert await aio.adict(async_gen()) == {1: 2, 3: 4, 5: 6} assert await aio.adict(empty_gen) == {} assert await aio.adict(empty_gen()) == {} python_utils-4.0.0/_python_utils_tests/test_aliases.py0000644000000045660000000000000170320ustar """The lightweight alias module must define the public type aliases without pulling in typing_extensions, so importers stay light. """ import os import subprocess import sys def test_aliases_do_not_import_typing_extensions() -> None: """Importing ``_aliases`` must not pull in typing_extensions.""" code = ( 'import sys, python_utils._aliases\n' "assert 'typing_extensions' not in sys.modules\n" ) result = subprocess.run( [sys.executable, '-c', code], capture_output=True, text=True, env={**os.environ, 'PYTHONPATH': os.pathsep.join(sys.path)}, ) assert result.returncode == 0, result.stderr def test_aliases_values() -> None: """Check the alias values and the ``__all__`` contents.""" from python_utils import _aliases assert _aliases.Number == (int | float) assert _aliases.delta_type == ( __import__('datetime').timedelta | int | float ) assert set(_aliases.__all__) == { 'Scope', 'OptionalScope', 'Number', 'DecimalNumber', 'ExceptionType', 'ExceptionsType', 'StringTypes', 'delta_type', 'timestamp_type', } def test_types_reexports_aliases_identically() -> None: """Re-export every ``_aliases`` name identically via ``types``.""" from python_utils import _aliases, types for name in _aliases.__all__: assert getattr(types, name) is getattr(_aliases, name), name def test_types_still_exposes_datetime_and_decimal() -> None: """Keep ``datetime`` and ``decimal`` as ``types`` module attributes.""" # Backwards compatibility: on 3.x these leaked as module attributes via # module-level imports, so `from python_utils.types import datetime` and # attribute access both worked. Moving the aliases into _aliases must not # drop them. import datetime import decimal from python_utils import types assert types.datetime is datetime assert types.decimal is decimal def test_types_still_exposes_typing_extensions_surface() -> None: """The ``types`` facade still re-exports ``Self``.""" # The facade must keep re-exporting typing_extensions (e.g. Self). # ``hasattr`` (not ``types.Self``) avoids basedpyright's # reportUnknownMemberType, since the wildcard re-export has no static type. from python_utils import types assert hasattr(types, 'Self') python_utils-4.0.0/_python_utils_tests/test_containers.py0000644000000041620000000000000175460ustar """Tests for the container types in ``python_utils.containers``.""" import pytest from python_utils import containers def test_unique_list_ignore() -> None: """Ignore duplicate appends and block duplicate slice sets.""" a: containers.UniqueList[int] = containers.UniqueList() a.append(1) a.append(1) assert a == [1] a = containers.UniqueList(*range(20)) with pytest.raises(RuntimeError): a[10:20:2] = [1, 2, 3, 4, 5] a[3] = 5 def test_unique_list_raise() -> None: """Raise on duplicates when ``on_duplicate='raise'``.""" a: containers.UniqueList[int] = containers.UniqueList( *range(20), on_duplicate='raise' ) with pytest.raises(ValueError): a[10:20:2] = [1, 2, 3, 4, 5] a[10:20:2] = [21, 22, 23, 24, 25] with pytest.raises(ValueError): a[3] = 5 del a[10] del a[5:15] def test_sliceable_deque() -> None: """Support indexing and extended slicing on the deque.""" d: containers.SliceableDeque[int] = containers.SliceableDeque(range(10)) assert d[0] == 0 assert d[-1] == 9 assert d[1:3] == [1, 2] assert d[1:3:2] == [1] assert d[1:3:-1] == [] assert d[3:1] == [] assert d[3:1:-1] == [3, 2] assert d[3:1:-2] == [3] with pytest.raises(ValueError): assert d[1:3:0] assert d[1:3:1] == [1, 2] assert d[1:3:2] == [1] assert d[1:3:-1] == [] def test_sliceable_deque_pop() -> None: """Pop by index and raise ``IndexError`` when out of range.""" d: containers.SliceableDeque[int] = containers.SliceableDeque(range(10)) assert d.pop() == 9 == 9 assert d.pop(0) == 0 with pytest.raises(IndexError): assert d.pop(100) with pytest.raises(IndexError): assert d.pop(2) with pytest.raises(IndexError): assert d.pop(-2) def test_sliceable_deque_eq() -> None: """Compare equal to list, tuple, set, and another deque.""" d: containers.SliceableDeque[int] = containers.SliceableDeque([1, 2, 3]) assert d == [1, 2, 3] assert d == (1, 2, 3) assert d == {1, 2, 3} assert d == d assert d == containers.SliceableDeque([1, 2, 3]) python_utils-4.0.0/_python_utils_tests/test_decorators.py0000644000000050200000000000000175400ustar """Tests for the decorators in ``python_utils.decorators``.""" import typing from unittest import mock import pytest from python_utils import decorators T = typing.TypeVar('T') @pytest.fixture def random(monkeypatch: pytest.MonkeyPatch) -> mock.MagicMock: """Patch ``decorators.random.random`` and return the mock.""" random_mock = mock.MagicMock() monkeypatch.setattr( 'python_utils.decorators.random.random', random_mock, raising=True ) return random_mock def test_sample_called(random: mock.MagicMock) -> None: """Call the wrapped function when the sampled roll passes.""" demo_function = mock.MagicMock() decorated = decorators.sample(0.5)(demo_function) random.return_value = 0.4 decorated() random.return_value = 0.0 decorated() args = [1, 2] kwargs = {'1': 1, '2': 2} decorated(*args, **kwargs) demo_function.assert_called_with(*args, **kwargs) assert demo_function.call_count == 3 def test_sample_not_called(random: mock.MagicMock) -> None: """Skip the wrapped function when the sampled roll fails.""" demo_function = mock.MagicMock() decorated = decorators.sample(0.5)(demo_function) random.return_value = 0.5 decorated() random.return_value = 1.0 decorated() assert demo_function.call_count == 0 class SomeClass: """A sample class with classmethods for wrapping tests.""" @classmethod def some_classmethod(cls, arg: T) -> T: """Return the argument unchanged (generic classmethod).""" return arg @classmethod def some_annotated_classmethod(cls, arg: int) -> int: """Return the integer argument unchanged.""" return arg def test_wraps_classmethod() -> None: """Forward calls through a wrapped classmethod.""" some_class = SomeClass() some_class.some_classmethod = mock.MagicMock() # type: ignore[method-assign] wrapped_method = decorators.wraps_classmethod(SomeClass.some_classmethod)( some_class.some_classmethod ) wrapped_method(123) some_class.some_classmethod.assert_called_with(123) def test_wraps_annotated_classmethod() -> None: """Forward calls through a wrapped annotated classmethod.""" some_class = SomeClass() some_class.some_annotated_classmethod = mock.MagicMock() # type: ignore[method-assign] wrapped_method = decorators.wraps_classmethod( SomeClass.some_annotated_classmethod )(some_class.some_annotated_classmethod) wrapped_method(123) some_class.some_annotated_classmethod.assert_called_with(123) python_utils-4.0.0/_python_utils_tests/test_generators.py0000644000000042020000000000000175450ustar """Tests for the batching helpers in ``python_utils.generators``.""" import asyncio import pytest import python_utils from python_utils import types @pytest.mark.asyncio async def test_abatcher() -> None: """Group an async count into fixed-size batches.""" async for batch in python_utils.abatcher(python_utils.acount(stop=9), 3): assert len(batch) == 3 async for batch in python_utils.abatcher(python_utils.acount(stop=2), 3): assert len(batch) == 2 @pytest.mark.asyncio async def test_abatcher_timed() -> None: """Group async items into batches by time interval.""" batches: types.List[types.List[int]] = [] async for batch in python_utils.abatcher( python_utils.acount(stop=10, delay=0.08), interval=0.1 ): batches.append(batch) assert batches == [[0, 1, 2], [3, 4], [5, 6], [7, 8], [9]] assert len(batches) == 5 @pytest.mark.asyncio async def test_abatcher_timed_with_timeout() -> None: """Respect timeouts and propagate errors while batching.""" async def generator() -> types.AsyncIterator[int]: """Yield items with sleeps to exercise batch timeouts.""" # Test if the timeout is respected yield 0 yield 1 await asyncio.sleep(0.11) # Test if the timeout is respected yield 2 yield 3 await asyncio.sleep(0.11) # Test if exceptions are handled correctly await asyncio.wait_for(asyncio.sleep(1), timeout=0.05) # Test if StopAsyncIteration is handled correctly yield 4 batcher = python_utils.abatcher(generator(), interval=0.1) assert await batcher.__anext__() == [0, 1] assert await batcher.__anext__() == [2, 3] with pytest.raises(asyncio.TimeoutError): await batcher.__anext__() with pytest.raises(StopAsyncIteration): await batcher.__anext__() def test_batcher() -> None: """Split an iterable into fixed-size batches.""" batch = [] for batch in python_utils.batcher(range(9), 3): assert len(batch) == 3 for batch in python_utils.batcher(range(4), 3): assert batch is not None assert len(batch) == 1 python_utils-4.0.0/_python_utils_tests/test_import.py0000644000000041250000000000000167120ustar """Tests for the import helpers in ``python_utils.import_``.""" from python_utils import import_, types def test_import_globals_relative_import() -> None: """Resolve relative imports across several levels.""" for i in range(-1, 5): relative_import(i) def relative_import(level: int) -> None: """Import ``.formatters`` relatively into a fake module dict.""" locals_: types.Dict[str, types.Any] = {} globals_ = {'__name__': 'python_utils.import_'} import_.import_global('.formatters', locals_=locals_, globals_=globals_) assert 'camel_to_underscore' in globals_ def test_import_globals_without_inspection() -> None: """Import a module without inspecting the caller frame.""" locals_: types.Dict[str, types.Any] = {} globals_: types.Dict[str, types.Any] = {'__name__': __name__} import_.import_global( 'python_utils.formatters', locals_=locals_, globals_=globals_ ) assert 'camel_to_underscore' in globals_ def test_import_globals_single_method() -> None: """Import only the single named attribute.""" locals_: types.Dict[str, types.Any] = {} globals_: types.Dict[str, types.Any] = {'__name__': __name__} import_.import_global( 'python_utils.formatters', ['camel_to_underscore'], locals_=locals_, globals_=globals_, ) assert 'camel_to_underscore' in globals_ def test_import_globals_with_inspection() -> None: """Infer the target globals from the caller frame.""" import_.import_global('python_utils.formatters') assert 'camel_to_underscore' in globals() def test_import_globals_missing_module() -> None: """Ignore ``ImportError`` for a missing module (locals_).""" import_.import_global( 'python_utils.spam', exceptions=ImportError, locals_=locals() ) assert 'camel_to_underscore' in globals() def test_import_locals_missing_module() -> None: """Ignore ``ImportError`` for a missing module (globals_).""" import_.import_global( 'python_utils.spam', exceptions=ImportError, globals_=globals() ) assert 'camel_to_underscore' in globals() python_utils-4.0.0/_python_utils_tests/test_import_footprint.py0000644000000100440000000000000210130ustar """Deterministic import-footprint regression gate. Each target module is imported in a clean subprocess; a per-target denylist of heavy modules must be absent from sys.modules afterward. This is the CI performance guard: it fails the moment an eager heavy import is reintroduced. """ import json import os import subprocess import sys import pytest # (import target, modules that must be ABSENT from sys.modules afterward) FOOTPRINT_CASES = [ ('python_utils', ('typing_extensions', 'asyncio')), ('python_utils.time', ('typing_extensions', 'asyncio')), ('python_utils.logger', ('typing_extensions',)), ('python_utils.converters', ('typing_extensions', 'asyncio')), ('python_utils.formatters', ('typing_extensions', 'asyncio')), ('python_utils.import_', ('typing_extensions', 'asyncio')), ('python_utils.terminal', ('typing_extensions',)), ('python_utils.containers', ('typing_extensions', 'asyncio')), ('python_utils.decorators', ('typing_extensions', 'asyncio')), ('python_utils.exceptions', ('typing_extensions', 'asyncio')), # aio, generators legitimately use asyncio; only typing_extensions denied. ('python_utils.aio', ('typing_extensions',)), ('python_utils.generators', ('typing_extensions',)), ] def _modules_after_import(target: str) -> set[str]: """Return ``sys.modules`` after importing ``target`` cleanly.""" code = ( f'import sys, {target}\n' 'import json\n' 'print(json.dumps(sorted(sys.modules)))\n' ) result = subprocess.run( [sys.executable, '-c', code], capture_output=True, text=True, env={**os.environ, 'PYTHONPATH': os.pathsep.join(sys.path)}, ) assert result.returncode == 0, result.stderr return set(json.loads(result.stdout)) @pytest.mark.parametrize(('target', 'denied'), FOOTPRINT_CASES) def test_import_footprint(target: str, denied: tuple[str, ...]) -> None: """Importing ``target`` must not pull in denied modules.""" present = _modules_after_import(target) leaked = [m for m in denied if m in present] assert not leaked, f'{target} eagerly imported {leaked}' def test_bare_import_module_count_under_budget() -> None: """Keep a bare import under the module-count budget.""" # Coarse bloat tripwire (denylist above is the real guard). Cap tightened # now that __version__ is lazy (importlib.metadata no longer pulled on bare # import). Bump only if a new Python version legitimately adds startup # modules. Measures modules ADDED by importing python_utils. added = len(_modules_after_import('python_utils')) - len( _modules_after_import('sys') ) assert added < 40, f'python_utils added {added} modules to sys.modules' def test_bare_import_does_not_pull_importlib_metadata() -> None: """A bare import must not import ``importlib.metadata``.""" # __version__ is resolved lazily; bare import must not call # importlib.metadata.version() (which drags in email/zipfile/json/...). present = _modules_after_import('python_utils') assert 'importlib.metadata' not in present def test_version_resolves_correctly() -> None: """Resolve ``__version__`` lazily to a non-empty string.""" import python_utils assert isinstance(python_utils.__version__, str) assert python_utils.__version__ # non-empty PUBLIC_CALLABLES_TO_INTROSPECT = [ ('python_utils.time', 'timeout_generator'), ('python_utils.time', 'aio_timeout_generator'), ('python_utils.time', 'format_time'), ('python_utils.converters', 'remap'), ('python_utils.converters', 'to_int'), ('python_utils.formatters', 'timesince'), ('python_utils.import_', 'import_global'), ] @pytest.mark.parametrize(('module', 'name'), PUBLIC_CALLABLES_TO_INTROSPECT) def test_get_type_hints_still_resolves(module: str, name: str) -> None: """Resolve type hints without ``NameError`` for public APIs.""" import importlib import typing obj = getattr(importlib.import_module(module), name) # Must not raise NameError now that type imports moved/changed. typing.get_type_hints(obj) python_utils-4.0.0/_python_utils_tests/test_lazy_imports.py0000644000000103730000000000000201360ustar """Tests for the lazy-import machinery that keeps `import python_utils` light (PEP 562 `__getattr__`/`__dir__` in the package, deferred ``asyncio`` in ``python_utils.time``). """ import collections.abc import os import subprocess import sys import pytest import python_utils def _run_clean(code: str) -> subprocess.CompletedProcess[str]: """Run ``code`` in a fresh interpreter and return the result.""" # Run in a fresh interpreter: the test session itself has long since # imported asyncio/typing_extensions, so in-process checks are useless. env = {**os.environ, 'PYTHONPATH': os.pathsep.join(sys.path)} return subprocess.run( [sys.executable, '-c', code], capture_output=True, text=True, env=env, ) def test_package_lazy_attribute_access() -> None: """Resolve submodule and exported names via ``__getattr__``.""" # Submodule access and exported-name access both resolve via __getattr__. aio = python_utils.aio assert python_utils.aio is aio # repeated access returns the cached module assert callable(python_utils.acount) assert isinstance(python_utils.__version__, str) missing = 'definitely_not_a_real_attribute' with pytest.raises(AttributeError): getattr(python_utils, missing) def test_bare_import_stays_light() -> None: """Keep a bare import free of asyncio and typing_extensions.""" # Importing the package must not eagerly pull in heavy/optional deps. result = _run_clean( 'import sys, python_utils\n' "assert 'asyncio' not in sys.modules, " "sorted(m for m in sys.modules if m.startswith('asyncio'))\n" "assert 'typing_extensions' not in sys.modules\n" ) assert result.returncode == 0, result.stderr def test_importing_time_submodule_avoids_asyncio() -> None: """Import ``python_utils.time`` without importing asyncio.""" # Importing python_utils.time for its synchronous helpers must not import # asyncio; the async helpers import it lazily inside their own bodies. result = _run_clean( 'import sys, python_utils.time\n' "assert 'asyncio' not in sys.modules, " "sorted(m for m in sys.modules if m.startswith('asyncio'))\n" ) assert result.returncode == 0, result.stderr def test_first_access_caches_into_module_dict() -> None: """Cache the first lazy access into the module dict.""" # PEP 562 __getattr__ runs once: the resolved object is cached in the # module namespace so subsequent lookups skip __getattr__ entirely. module = python_utils.time assert python_utils.__dict__['time'] is module func = python_utils.format_time assert python_utils.__dict__['format_time'] is func def test_dir_lists_lazy_submodules() -> None: """List lazy submodules and ``__all__`` names via ``dir``.""" # Lazy submodules that are not in __all__ (e.g. ``containers`` and # ``exceptions``) must still be discoverable via ``dir``; tools such as # ``import_global`` intersect requested names with ``dir(module)``. names = set(dir(python_utils)) assert {'containers', 'exceptions'} <= names assert set(python_utils.__all__) <= names def test_star_import_resolves_all_names() -> None: """Bind every ``__all__`` name via a package star-import.""" # `from python_utils import *` goes through __getattr__ for every name in # __all__; none may raise AttributeError, and all must be bound. namespace: dict[str, object] = {} exec('from python_utils import *', namespace) # noqa: S102 missing = set(python_utils.__all__) - set(namespace) assert not missing @pytest.mark.asyncio async def test_aio_timeout_generator_default_iterable() -> None: """Default the iterable to ``aio.acount`` when omitted.""" # With no iterable the generator defaults to ``aio.acount`` -- exercising # the lazy ``aio``/``asyncio`` import and the None-resolution branch. count = 0 generator: collections.abc.AsyncGenerator[object, None] = ( python_utils.aio_timeout_generator(timeout=0.05, interval=0.0) ) async for _ in generator: count += 1 if count >= 2: break assert count == 2 # Sanity: the re-exported type alias still resolves through the package. assert python_utils.types.AsyncGenerator is not None python_utils-4.0.0/_python_utils_tests/test_logger.py0000644000000010450000000000000166550ustar # mypy: disable-error-code=misc """Tests for the loguru mixin in ``python_utils.loguru``.""" import pytest from python_utils import loguru pytest.importorskip('loguru') def test_logurud() -> None: """Expose all loguru log-level methods on a subclass.""" class MyClass(loguru.Logurud): pass my_class = MyClass() my_class.debug('debug') my_class.info('info') my_class.warning('warning') my_class.error('error') my_class.critical('critical') my_class.exception('exception') my_class.log(0, 'log') python_utils-4.0.0/_python_utils_tests/test_python_utils.py0000644000000006270000000000000201440ustar """Tests for the package metadata in ``python_utils.__about__``.""" from python_utils import __about__ def test_definitions() -> None: """Ensure the required metadata fields are defined.""" # The setup.py requires this so we better make sure they exist :) assert __about__.__version__ assert __about__.__author__ assert __about__.__author_email__ assert __about__.__description__ python_utils-4.0.0/_python_utils_tests/test_time.py0000644000000136260000000000000163440ustar """Tests for the timeout generators in ``python_utils.time``.""" import asyncio import datetime import itertools import pytest import python_utils from python_utils import types @pytest.mark.parametrize( 'timeout,interval,interval_multiplier,maximum_interval,iterable,result', [ (0.2, 0.1, 0.4, 0.2, python_utils.acount, 2), (0.3, 0.1, 0.4, 0.2, python_utils.acount(), 3), (0.3, 0.06, 1.0, None, python_utils.acount, 5), ( datetime.timedelta(seconds=0.1), datetime.timedelta(seconds=0.06), 2.0, datetime.timedelta(seconds=0.1), python_utils.acount, 2, ), ], ) @pytest.mark.asyncio async def test_aio_timeout_generator( timeout: float, interval: float, interval_multiplier: float, maximum_interval: float, iterable: types.AsyncIterable[types.Any], result: int, ) -> None: """Stop the async generator near the configured timeout.""" i = None async for i in python_utils.aio_timeout_generator( timeout, interval, iterable, maximum_interval=maximum_interval ): pass assert i == result @pytest.mark.parametrize( 'timeout,interval,interval_multiplier,maximum_interval,iterable,result', [ (0.1, 0.06, 0.5, 0.1, 'abc', 'c'), (0.1, 0.07, 0.5, 0.1, itertools.count, 2), (0.1, 0.07, 0.5, 0.1, itertools.count(), 2), (0.1, 0.06, 1.0, None, 'abc', 'c'), ( datetime.timedelta(seconds=0.1), datetime.timedelta(seconds=0.06), 2.0, datetime.timedelta(seconds=0.1), itertools.count, 2, ), ], ) def test_timeout_generator( timeout: float, interval: float, interval_multiplier: float, maximum_interval: float, iterable: types.Union[ str, types.Iterable[types.Any], types.Callable[..., types.Iterable[types.Any]], ], result: int, ) -> None: """Stop the sync generator near the configured timeout.""" i = None for i in python_utils.timeout_generator( timeout=timeout, interval=interval, interval_multiplier=interval_multiplier, iterable=iterable, maximum_interval=maximum_interval, ): assert i is not None assert i == result @pytest.mark.asyncio async def test_aio_generator_timeout_detector() -> None: """Raise or exit on per-item and total timeouts.""" # Make pyright happy i = None async def generator() -> types.AsyncGenerator[int, None]: """Yield 0-9 with increasing sleeps between items.""" for i in range(10): await asyncio.sleep(i / 20.0) yield i detector = python_utils.aio_generator_timeout_detector # Test regular timeout with reraise with pytest.raises(asyncio.TimeoutError): async for i in detector(generator(), 0.25): pass # Test regular timeout with clean exit async for i in detector(generator(), 0.25, on_timeout=None): pass assert i == 4 # Test total timeout with reraise with pytest.raises(asyncio.TimeoutError): async for i in detector(generator(), total_timeout=0.5): pass # Test total timeout with clean exit async for i in detector(generator(), total_timeout=0.5, on_timeout=None): pass assert i == 4 # Test stop iteration async for i in detector(generator(), on_timeout=None): pass @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_reraise() -> None: """Reraise ``TimeoutError`` on a per-item timeout.""" # Test regular timeout with reraise @python_utils.aio_generator_timeout_detector_decorator(timeout=0.05) async def generator_timeout() -> types.AsyncGenerator[int, None]: """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i with pytest.raises(asyncio.TimeoutError): async for _ in generator_timeout(): pass @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_clean_exit() -> None: """Exit cleanly when ``on_timeout`` is ``None``.""" # Make pyright happy i = None # Test regular timeout with clean exit @python_utils.aio_generator_timeout_detector_decorator( timeout=0.05, on_timeout=None ) async def generator_clean() -> types.AsyncGenerator[int, None]: """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i async for i in generator_clean(): pass assert i == 4 @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_reraise_total() -> ( None ): """Reraise ``TimeoutError`` on a total timeout.""" # Test total timeout with reraise @python_utils.aio_generator_timeout_detector_decorator(total_timeout=0.1) async def generator_reraise() -> types.AsyncGenerator[int, None]: """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i with pytest.raises(asyncio.TimeoutError): async for _ in generator_reraise(): pass @pytest.mark.asyncio async def test_aio_generator_timeout_detector_decorator_clean_total() -> None: """Exit cleanly on total timeout when ``on_timeout`` is ``None``.""" # Make pyright happy i = None # Test total timeout with clean exit @python_utils.aio_generator_timeout_detector_decorator( total_timeout=0.1, on_timeout=None ) async def generator_clean_total() -> types.AsyncGenerator[int, None]: """Yield with increasing delays to trip the timeout.""" for i in range(10): await asyncio.sleep(i / 100.0) yield i async for i in generator_clean_total(): pass assert i == 4 python_utils-4.0.0/pyproject.toml0000644000000122270000000000000125630ustar [build-system] requires = ['uv_build>=0.11,<0.12'] build-backend = 'uv_build' [tool.uv.build-backend] # Flat layout: the package dir sits at the repo root. module-root = '' module-name = 'python_utils' # Keep the tests and tox config in the sdist (parity with the old MANIFEST.in) # so downstream packagers can build and test from the source distribution. source-include = ['_python_utils_tests/**/*.py', 'tox.ini'] [project] name = 'python-utils' version = '4.0.0' # static; the single source of truth. Bump via `uv version`. description = 'Python Utils is a module with some convenient utilities not included with the standard Python install' readme = 'README.md' requires-python = '>=3.10' license = 'BSD-3-Clause' license-files = ['LICENSE'] authors = [{ name = 'Rick van Hattem', email = 'wolph@wol.ph' }] keywords = [ 'utils', 'utilities', 'helpers', 'typing', 'typed', 'async', 'lazy-imports', ] classifiers = [ 'Development Status :: 5 - Production/Stable', 'Intended Audience :: Developers', 'Operating System :: OS Independent', 'Programming Language :: Python :: 3', 'Programming Language :: Python :: 3.10', 'Programming Language :: Python :: 3.11', 'Programming Language :: Python :: 3.12', 'Programming Language :: Python :: 3.13', 'Programming Language :: Python :: 3.14', 'Programming Language :: Python :: 3 :: Only', 'Programming Language :: Python :: Implementation :: CPython', 'Programming Language :: Python :: Implementation :: PyPy', 'Typing :: Typed', ] dependencies = ['typing_extensions>3.10.0.2'] [project.urls] Homepage = 'https://github.com/WoLpH/python-utils' Documentation = 'https://python-utils.readthedocs.io/' Repository = 'https://github.com/WoLpH/python-utils' Changelog = 'https://github.com/WoLpH/python-utils/releases' [project.optional-dependencies] loguru = ['loguru'] docs = ['sphinx', 'furo', 'myst-parser'] tests = ['pytest', 'pytest-cov', 'pytest-asyncio', 'loguru', 'blessings'] [dependency-groups] docs = ['sphinx', 'furo', 'myst-parser'] test = ['pytest', 'pytest-cov', 'pytest-asyncio', 'loguru', 'blessings'] dev = [ 'ruff', 'mypy', 'basedpyright', 'pyrefly', 'codespell', 'lefthook', { include-group = 'test' }, { include-group = 'docs' }, ] tox = ['tox>=4', 'tox-uv>=1', 'tox-gh-actions'] # --- Type checking (mypy + basedpyright + pyrefly, all strict) -------------- [tool.mypy] python_version = '3.10' strict = true check_untyped_defs = true files = ['python_utils', '_python_utils_tests'] [[tool.mypy.overrides]] module = '_python_utils_tests.*' disallow_untyped_defs = false [[tool.mypy.overrides]] # terminal.py is OS-specific (windll / blessings / IPython / fcntl.ioctl); # excluded from all three checkers for consistency. module = 'python_utils.terminal' ignore_errors = true [tool.pyright] # basedpyright reads the [tool.pyright] table (it is a pyright superset). typeCheckingMode = 'strict' pythonVersion = '3.10' include = ['python_utils', '_python_utils_tests'] exclude = ['**/__pycache__', '.tox', '.venv', 'build', 'dist', 'docs'] # The terminal file is very OS specific and dependent on imports so we're # skipping it from type checking. ignore = ['python_utils/terminal.py'] reportMissingTypeStubs = false [tool.pyrefly] project-includes = ['python_utils', '_python_utils_tests'] # terminal.py is OS-specific (windll / blessings / fcntl.ioctl) and pyrefly # does not honour `# type: ignore` / `# pyright:` comments, so exclude it. project-excludes = ['python_utils/terminal.py', '**/__pycache__'] python-version = '3.10' # --- Tests & coverage ------------------------------------------------------- [tool.pytest.ini_options] testpaths = ['python_utils', '_python_utils_tests'] python_files = ['python_utils/*.py', '_python_utils_tests/*.py'] addopts = [ '--strict-config', '--strict-markers', '-ra', '--doctest-modules', '--cov=python_utils', '--cov-report=term-missing', ] doctest_optionflags = ['ALLOW_UNICODE', 'ALLOW_BYTES'] asyncio_mode = 'strict' xfail_strict = true filterwarnings = [ 'error', # import_global() does dynamic relative imports; ImportWarning is silenced # by Python's default filters anyway and is benign here. 'ignore::ImportWarning', ] [tool.coverage.run] branch = true parallel = true # unique data file per run so matrix cells can be combined source = ['python_utils'] [tool.coverage.paths] # Map per-cell data files back onto the source tree when combining. source = ['python_utils', '*/python_utils'] [tool.coverage.report] fail_under = 100 show_missing = true exclude_also = [ 'pragma: no cover', 'if typing.TYPE_CHECKING:', 'if types.TYPE_CHECKING:', 'if __name__ == .__main__.:', 'if 0:', 'raise AssertionError', 'raise NotImplementedError', '@(abc\.)?abstractmethod', '@(typing\.|types\.)?overload', 'class .*\bProtocol\):', ] [tool.coverage.html] directory = 'htmlcov' [tool.codespell] skip = '*.lock,*.svg,*.egg-info,./.git,./.tox,./.venv,./.ruff_cache,./.mypy_cache,./.pytest_cache,./build,./dist,./htmlcov,./docs/_build,*/__pycache__' # `acount` is an intentional public API name (async count). ignore-words-list = 'acount' check-hidden = true python_utils-4.0.0/python_utils0000755000000000000000000000000123255ustar python_utils-4.0.0/python_utils/__about__.py0000644000000024770000000000000146760ustar """ This module contains metadata about the `python-utils` package. Attributes: __package_name__ (str): The name of the package. __author__ (str): The author of the package. __author_email__ (str): The email of the author. __description__ (str): A brief description of the package. __url__ (str): The URL of the package's repository. __version__ (str): The current version, read from the installed metadata. """ from __future__ import annotations from importlib import metadata #: Distribution name as published on PyPI. __package_name__: str = 'python-utils' #: Primary author's name. __author__: str = 'Rick van Hattem' #: Primary author's contact email. __author_email__: str = 'Wolph@wol.ph' #: One-line description of the package. __description__: str = ( 'Python Utils is a module with some convenient utilities not included ' 'with the standard Python install' ) #: Canonical project/repository URL. __url__: str = 'https://github.com/WoLpH/python-utils' try: # `[project].version` in pyproject.toml is the single source of truth; # read it back at runtime so the two never drift. __version__: str = metadata.version(__package_name__) except metadata.PackageNotFoundError: # pragma: no cover # Not installed (e.g. running straight from a source checkout). __version__ = '0.0.0' python_utils-4.0.0/python_utils/__init__.py0000644000000135060000000000000145220ustar """ This module initializes the `python_utils` package by importing various submodules and functions. Imports are performed lazily (PEP 562): nothing is imported when you ``import python_utils``; each submodule/function is loaded on first access. This keeps ``import python_utils`` cheap and, in particular, avoids eagerly importing ``asyncio`` (via the async helpers) for consumers that only need the synchronous utilities. Submodules:: aio converters decorators formatters generators import_ logger terminal time types Functions:: acount remap scale_1024 to_float to_int to_str to_unicode listify set_attributes raise_exception reraise camel_to_underscore timesince abatcher batcher import_global get_terminal_size aio_generator_timeout_detector aio_generator_timeout_detector_decorator aio_timeout_generator delta_to_seconds delta_to_seconds_or_none format_time timedelta_to_seconds timeout_generator Classes:: CastedDict LazyCastedDict UniqueList Logged LoggerBase """ import importlib as _importlib import typing as _typing if _typing.TYPE_CHECKING: # pragma: no cover # Eager imports for type checkers only; the runtime equivalents are loaded # lazily by ``__getattr__`` below. Names appear in ``__all__`` so they are # treated as re-exports (not unused imports). from . import ( aio, converters, decorators, formatters, generators, import_, logger, terminal, time, types, ) from .__about__ import __version__ from .aio import acount from .containers import CastedDict, LazyCastedDict, UniqueList from .converters import ( remap, scale_1024, to_float, to_int, to_str, to_unicode, ) from .decorators import listify, set_attributes from .exceptions import raise_exception, reraise from .formatters import camel_to_underscore, timesince from .generators import abatcher, batcher from .import_ import import_global from .logger import Logged, LoggerBase from .terminal import get_terminal_size from .time import ( aio_generator_timeout_detector, aio_generator_timeout_detector_decorator, aio_timeout_generator, delta_to_seconds, delta_to_seconds_or_none, format_time, timedelta_to_seconds, timeout_generator, ) #: Submodules that can be accessed as ``python_utils.``. _SUBMODULES: frozenset[str] = frozenset( { 'aio', 'containers', 'converters', 'decorators', 'exceptions', 'formatters', 'generators', 'import_', 'logger', 'terminal', 'time', 'types', } ) #: Exported name -> submodule it lives in. _NAME_TO_MODULE: dict[str, str] = { '__version__': '__about__', 'acount': 'aio', 'CastedDict': 'containers', 'LazyCastedDict': 'containers', 'UniqueList': 'containers', 'remap': 'converters', 'scale_1024': 'converters', 'to_float': 'converters', 'to_int': 'converters', 'to_str': 'converters', 'to_unicode': 'converters', 'listify': 'decorators', 'set_attributes': 'decorators', 'raise_exception': 'exceptions', 'reraise': 'exceptions', 'camel_to_underscore': 'formatters', 'timesince': 'formatters', 'abatcher': 'generators', 'batcher': 'generators', 'import_global': 'import_', 'Logged': 'logger', 'LoggerBase': 'logger', 'get_terminal_size': 'terminal', 'aio_generator_timeout_detector': 'time', 'aio_generator_timeout_detector_decorator': 'time', 'aio_timeout_generator': 'time', 'delta_to_seconds': 'time', 'delta_to_seconds_or_none': 'time', 'format_time': 'time', 'timedelta_to_seconds': 'time', 'timeout_generator': 'time', } def __getattr__(name: str) -> _typing.Any: """Lazily import a submodule or exported name on first access (PEP 562). Args: name: Attribute requested on the ``python_utils`` package. Returns: The imported submodule or object. It is cached in ``globals()`` so this hook runs only once per name. Raises: AttributeError: If ``name`` is not a known submodule or export. """ if name in _SUBMODULES: module = _importlib.import_module(f'.{name}', __name__) elif name in _NAME_TO_MODULE: module = _importlib.import_module( f'.{_NAME_TO_MODULE[name]}', __name__ ) value = getattr(module, name) globals()[name] = value # cache so __getattr__ runs only once return value else: raise AttributeError(f'module {__name__!r} has no attribute {name!r}') globals()[name] = module return module def __dir__() -> list[str]: """List all eager and lazily-available names (for tab-completion).""" return sorted( set(globals()) | set(__all__) | _SUBMODULES | set(_NAME_TO_MODULE) ) __all__ = [ 'CastedDict', 'LazyCastedDict', 'Logged', 'LoggerBase', 'UniqueList', '__version__', 'abatcher', 'acount', 'aio', 'aio_generator_timeout_detector', 'aio_generator_timeout_detector_decorator', 'aio_timeout_generator', 'batcher', 'camel_to_underscore', 'converters', 'decorators', 'delta_to_seconds', 'delta_to_seconds_or_none', 'format_time', 'formatters', 'generators', 'get_terminal_size', 'import_', 'import_global', 'listify', 'logger', 'raise_exception', 'remap', 'reraise', 'scale_1024', 'set_attributes', 'terminal', 'time', 'timedelta_to_seconds', 'timeout_generator', 'timesince', 'to_float', 'to_int', 'to_str', 'to_unicode', 'types', ] python_utils-4.0.0/python_utils/_aliases.py0000644000000031510000000000000145360ustar """Lightweight, stdlib-only type aliases shared across python_utils. These live here (rather than in ``python_utils.types``) so internal modules can import them without dragging in ``typing_extensions``. ``python_utils.types`` re-exports everything defined here, so the public names are unchanged. """ from __future__ import annotations import datetime import decimal from typing import Any __all__ = [ 'DecimalNumber', 'ExceptionType', 'ExceptionsType', 'Number', 'OptionalScope', 'Scope', 'StringTypes', 'delta_type', 'timestamp_type', ] #: A namespace mapping, e.g. ``locals()``/``globals()`` (name -> value). Scope = dict[str, Any] #: A :data:`Scope`, or ``None`` when no namespace is supplied. OptionalScope = Scope | None #: Any plain (non-decimal) number: an ``int`` or a ``float``. Number = int | float #: A :data:`Number` or a :class:`decimal.Decimal`, for precise arithmetic. DecimalNumber = Number | decimal.Decimal #: An exception class (not an instance), e.g. ``ValueError``. ExceptionType = type[Exception] #: One exception class or a tuple of them, as accepted by ``except``. ExceptionsType = tuple[ExceptionType, ...] | ExceptionType #: Text-like data: ``str`` or ``bytes``. StringTypes = str | bytes #: A time interval expressed as a ``timedelta`` or a number of seconds. delta_type = datetime.timedelta | int | float #: Anything :func:`~python_utils.time.format_time` can render: a duration, a #: date/datetime, a numeric/str timestamp, or ``None``. timestamp_type = ( datetime.timedelta | datetime.date | datetime.datetime | str | int | float | None ) python_utils-4.0.0/python_utils/aio.py0000644000000076370000000000000135430ustar """Asyncio equivalents of common synchronous helpers. These bring ``itertools``-style ergonomics to ``async for``: ``acount`` is an async counter, while ``acontainer`` and ``adict`` collect an async iterable into a concrete container. """ import asyncio import collections.abc import itertools import typing #: Numeric type (``int`` or ``float``) produced by ``acount``. _N = typing.TypeVar('_N', int, float) #: Element type of the async iterables being consumed. _T = typing.TypeVar('_T') #: Key type when collecting an async iterable of pairs into a dict. _K = typing.TypeVar('_K') #: Value type when collecting an async iterable of pairs into a dict. _V = typing.TypeVar('_V') async def acount( start: _N = 0, step: _N = 1, delay: float = 0, stop: _N | None = None, ) -> collections.abc.AsyncIterator[_N]: """Async equivalent of ``itertools.count()``. Counts from ``start`` in steps of ``step``, sleeping ``delay`` seconds between values, and stops once ``stop`` (when given) is reached. Args: start: First value to yield. step: Amount added between successive values. delay: Seconds to ``asyncio.sleep`` between yields. stop: Exclusive upper bound; ``None`` counts forever. Yields: The successive counter values. >>> async def demo(): ... return [i async for i in acount(stop=3)] >>> asyncio.run(demo()) [0, 1, 2] """ for item in itertools.count(start, step): # pragma: no branch if stop is not None and item >= stop: break yield item await asyncio.sleep(delay) @typing.overload async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: type[tuple[_T, ...]], ) -> tuple[_T, ...]: """Overload: collect the async iterable into a ``tuple``.""" @typing.overload async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: type[list[_T]] = list, ) -> list[_T]: """Overload: collect the async iterable into a ``list`` (the default).""" @typing.overload async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: type[set[_T]], ) -> set[_T]: """Overload: collect the async iterable into a ``set``.""" async def acontainer( iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]], container: collections.abc.Callable[ [collections.abc.Iterable[_T]], collections.abc.Collection[_T] ] = list, ) -> collections.abc.Collection[_T]: """ Asyncio version of list()/set()/tuple()/etc() using an async for loop. So instead of doing `[item async for item in iterable]` you can do `await acontainer(iterable)`. """ iterable_: collections.abc.AsyncIterable[_T] if callable(iterable): iterable_ = iterable() else: iterable_ = iterable items: list[_T] = [item async for item in iterable_] return container(items) async def adict( iterable: collections.abc.AsyncIterable[tuple[_K, _V]] | collections.abc.Callable[ ..., collections.abc.AsyncIterable[tuple[_K, _V]] ], container: collections.abc.Callable[ [collections.abc.Iterable[tuple[_K, _V]]], collections.abc.Mapping[_K, _V], ] = dict, ) -> collections.abc.Mapping[_K, _V]: """ Asyncio version of dict() using an async for loop. So instead of doing `{key: value async for key, value in iterable}` you can do `await adict(iterable)`. """ iterable_: collections.abc.AsyncIterable[tuple[_K, _V]] if callable(iterable): iterable_ = iterable() else: iterable_ = iterable items: list[tuple[_K, _V]] = [item async for item in iterable_] return container(items) python_utils-4.0.0/python_utils/containers.py0000644000000465110000000000000151320ustar """ This module provides custom container classes with enhanced functionality. Classes:: CastedDictBase: Abstract base class for dictionaries that cast keys and values. CastedDict: Dictionary that casts keys and values to specified types. LazyCastedDict: Dictionary that lazily casts values to specified types upon access. UniqueList: List that only allows unique values, with configurable behavior on duplicates. SliceableDeque: Deque that supports slicing and enhanced equality checks. Type Aliases:: KT: Type variable for dictionary keys. VT: Type variable for dictionary values. DT: Type alias for a dictionary with keys of type KT and values of type VT. KT_cast: Type alias for a callable that casts dictionary keys. VT_cast: Type alias for a callable that casts dictionary values. HT: Type variable for hashable values in UniqueList. T: Type variable for generic types. DictUpdateArgs: Union type for arguments that can be used to update a dictionary. OnDuplicate: Literal type for handling duplicate values in UniqueList. Usage:: - CastedDict and LazyCastedDict can be used to create dictionaries with automatic type casting. - UniqueList ensures all elements are unique and can raise an error on duplicates. - SliceableDeque extends deque with slicing support and enhanced equality checks. Examples: >>> d = CastedDict(int, int) >>> d[1] = 2 >>> d['3'] = '4' >>> d.update({'5': '6'}) >>> d.update([('7', '8')]) >>> d {1: 2, 3: 4, 5: 6, 7: 8} >>> l = UniqueList(1, 2, 3) >>> l.append(4) >>> l.append(4) >>> l.insert(0, 4) >>> l.insert(0, 5) >>> l[1] = 10 >>> l [5, 10, 2, 3, 4] >>> d = SliceableDeque([1, 2, 3, 4, 5]) >>> d[1:4] SliceableDeque([2, 3, 4]) """ # pyright: reportIncompatibleMethodOverride=false import abc import collections import collections.abc import typing if typing.TYPE_CHECKING: import _typeshed # noqa: F401 #: A type alias for a type that can be used as a key in a dictionary. KT = typing.TypeVar('KT') #: A type alias for a type that can be used as a value in a dictionary. VT = typing.TypeVar('VT') #: A type alias for a dictionary with keys of type KT and values of type VT. DT = dict[KT, VT] #: A type alias for the casted type of a dictionary key. KT_cast = collections.abc.Callable[..., KT] | None #: A type alias for the casted type of a dictionary value. VT_cast = collections.abc.Callable[..., VT] | None #: A type alias for the hashable values of the `UniqueList` HT = typing.TypeVar('HT', bound=collections.abc.Hashable) #: A type alias for a regular generic type T = typing.TypeVar('T') #: Argument shapes accepted when updating a casted dict: a mapping, an iterable #: of key/value pairs, an iterable of mappings, or a keys-and-getitem object. # Kept as `typing.Union` (not PEP 604 `|`): one member is a string forward # reference, and `|` evaluates its operands eagerly, raising `TypeError` on a # `str` operand at runtime. `typing.Union` accepts it as a lazy `ForwardRef`. DictUpdateArgs = typing.Union[ collections.abc.Mapping[KT, VT], collections.abc.Iterable[tuple[KT, VT]], collections.abc.Iterable[collections.abc.Mapping[KT, VT]], '_typeshed.SupportsKeysAndGetItem[KT, VT]', ] #: Policy for ``UniqueList`` duplicates: silently ``'ignore'`` or ``'raise'``. OnDuplicate = typing.Literal['ignore', 'raise'] class CastedDictBase(dict[KT, VT], abc.ABC): """ Abstract base class for dictionaries that cast keys and values. Attributes: _key_cast (KT_cast[KT]): Callable to cast dictionary keys. _value_cast (VT_cast[VT]): Callable to cast dictionary values. Methods:: __init__(key_cast: KT_cast[KT] = None, value_cast: VT_cast[VT] = None, *args: DictUpdateArgs[KT, VT], **kwargs: VT) -> None: Initializes the dictionary with optional key and value casting callables. update(*args: DictUpdateArgs[typing.Any, typing.Any], **kwargs: typing.Any) -> None: Updates the dictionary with the given arguments. __setitem__(key: typing.Any, value: typing.Any) -> None: Sets the item in the dictionary, casting the key if a key cast callable is provided. """ _key_cast: KT_cast[KT] _value_cast: VT_cast[VT] def __init__( self, key_cast: KT_cast[KT] = None, value_cast: VT_cast[VT] = None, *args: DictUpdateArgs[KT, VT], **kwargs: VT, ) -> None: """ Initializes the CastedDictBase with optional key and value casting callables. Args: key_cast (KT_cast[KT], optional): Callable to cast dictionary keys. Defaults to None. value_cast (VT_cast[VT], optional): Callable to cast dictionary values. Defaults to None. *args (DictUpdateArgs[KT, VT]): Arguments to initialize the dictionary. **kwargs (VT): Keyword arguments to initialize the dictionary. """ self._value_cast = value_cast self._key_cast = key_cast self.update(*args, **kwargs) def update( self, *args: DictUpdateArgs[typing.Any, typing.Any], **kwargs: typing.Any, ) -> None: """ Updates the dictionary with the given arguments. Args: *args (DictUpdateArgs[typing.Any, typing.Any]): Arguments to update the dictionary. **kwargs (typing.Any): Keyword arguments to update the dictionary. """ if args: kwargs.update(*args) if kwargs: for key, value in kwargs.items(): self[key] = value def __setitem__(self, key: typing.Any, value: typing.Any) -> None: """ Sets the item in the dictionary, casting the key if a key cast callable is provided. Args: key (typing.Any): The key to set in the dictionary. value (typing.Any): The value to set in the dictionary. """ if self._key_cast is not None: key = self._key_cast(key) return super().__setitem__(key, value) class CastedDict(CastedDictBase[KT, VT]): """ Custom dictionary that casts keys and values to the specified types. Note that you can specify the types for mypy and type hinting with: CastedDict[int, int](int, int) >>> d: CastedDict[int, int] = CastedDict(int, int) >>> d[1] = 2 >>> d['3'] = '4' >>> d.update({'5': '6'}) >>> d.update([('7', '8')]) >>> d {1: 2, 3: 4, 5: 6, 7: 8} >>> list(d.keys()) [1, 3, 5, 7] >>> list(d) [1, 3, 5, 7] >>> list(d.values()) [2, 4, 6, 8] >>> list(d.items()) [(1, 2), (3, 4), (5, 6), (7, 8)] >>> d[3] 4 # Casts are optional and can be disabled by passing None as the cast >>> d = CastedDict() >>> d[1] = 2 >>> d['3'] = '4' >>> d.update({'5': '6'}) >>> d.update([('7', '8')]) >>> d {1: 2, '3': '4', '5': '6', '7': '8'} """ def __setitem__(self, key: typing.Any, value: typing.Any) -> None: """Cast ``value`` (if a value cast is set) and store it under ``key``. The key itself is cast by ``CastedDictBase.__setitem__`` when a key cast is configured. """ if self._value_cast is not None: value = self._value_cast(value) super().__setitem__(key, value) class LazyCastedDict(CastedDictBase[KT, VT]): """ Custom dictionary that casts keys and lazily casts values to the specified types. Note that the values are cast only when they are accessed and are not cached between executions. Note that you can specify the types for mypy and type hinting with: LazyCastedDict[int, int](int, int) >>> d: LazyCastedDict[int, int] = LazyCastedDict(int, int) >>> d[1] = 2 >>> d['3'] = '4' >>> d.update({'5': '6'}) >>> d.update([('7', '8')]) >>> d {1: 2, 3: '4', 5: '6', 7: '8'} >>> list(d.keys()) [1, 3, 5, 7] >>> list(d) [1, 3, 5, 7] >>> list(d.values()) [2, 4, 6, 8] >>> list(d.items()) [(1, 2), (3, 4), (5, 6), (7, 8)] >>> d[3] 4 # Casts are optional and can be disabled by passing None as the cast >>> d = LazyCastedDict() >>> d[1] = 2 >>> d['3'] = '4' >>> d.update({'5': '6'}) >>> d.update([('7', '8')]) >>> d {1: 2, '3': '4', '5': '6', '7': '8'} >>> list(d.keys()) [1, '3', '5', '7'] >>> list(d.values()) [2, '4', '6', '8'] >>> list(d.items()) [(1, 2), ('3', '4'), ('5', '6'), ('7', '8')] >>> d['3'] '4' """ def __setitem__(self, key: typing.Any, value: typing.Any) -> None: """ Sets the item in the dictionary, casting the key if a key cast callable is provided. Args: key (typing.Any): The key to set in the dictionary. value (typing.Any): The value to set in the dictionary. """ if self._key_cast is not None: key = self._key_cast(key) super().__setitem__(key, value) def __getitem__(self, key: typing.Any) -> VT: """ Gets the item from the dictionary, casting the value if a value cast callable is provided. Args: key (typing.Any): The key to get from the dictionary. Returns: VT: The value from the dictionary. """ if self._key_cast is not None: key = self._key_cast(key) value = super().__getitem__(key) if self._value_cast is not None: value = self._value_cast(value) return value def items( # type: ignore[override] self, ) -> collections.abc.Generator[tuple[KT, VT], None, None]: """ Returns a generator of the dictionary's items, casting the values if a value cast callable is provided. Yields: Generator[tuple[KT, VT], None, None]: A generator of the dictionary's items. """ if self._value_cast is None: yield from super().items() else: for key, value in super().items(): yield key, self._value_cast(value) def values(self) -> collections.abc.Generator[VT, None, None]: # type: ignore[override] """ Returns a generator of the dictionary's values, casting the values if a value cast callable is provided. Yields: Generator[VT, None, None]: A generator of the dictionary's values. """ if self._value_cast is None: yield from super().values() else: for value in super().values(): yield self._value_cast(value) class UniqueList(list[HT]): """ A list that only allows unique values. Duplicate values are ignored by default, but can be configured to raise an exception instead. >>> l = UniqueList(1, 2, 3) >>> l.append(4) >>> l.append(4) >>> l.insert(0, 4) >>> l.insert(0, 5) >>> l[1] = 10 >>> l [5, 10, 2, 3, 4] >>> l = UniqueList(1, 2, 3, on_duplicate='raise') >>> l.append(4) >>> l.append(4) Traceback (most recent call last): ... ValueError: Duplicate value: 4 >>> l.insert(0, 4) Traceback (most recent call last): ... ValueError: Duplicate value: 4 >>> 4 in l True >>> l[0] 1 >>> l[1] = 4 Traceback (most recent call last): ... ValueError: Duplicate value: 4 """ _set: set[HT] def __init__( self, *args: HT, on_duplicate: OnDuplicate = 'ignore', ): """ Initializes the UniqueList with optional duplicate handling behavior. Args: *args (HT): Initial values for the list. on_duplicate (OnDuplicate, optional): Behavior on duplicates. Defaults to 'ignore'. """ self.on_duplicate = on_duplicate self._set = set() super().__init__() for arg in args: self.append(arg) def insert(self, index: typing.SupportsIndex, value: HT) -> None: """ Inserts a value at the specified index, ensuring uniqueness. Args: index (typing.SupportsIndex): The index to insert the value at. value (HT): The value to insert. Raises: ValueError: If the value is a duplicate and `on_duplicate` is set to 'raise'. """ if value in self._set: if self.on_duplicate == 'raise': raise ValueError(f'Duplicate value: {value}') else: return self._set.add(value) super().insert(index, value) def append(self, value: HT) -> None: """ Appends a value to the list, ensuring uniqueness. Args: value (HT): The value to append. Raises: ValueError: If the value is a duplicate and `on_duplicate` is set to 'raise'. """ if value in self._set: if self.on_duplicate == 'raise': raise ValueError(f'Duplicate value: {value}') else: return self._set.add(value) super().append(value) def __contains__(self, item: HT) -> bool: # type: ignore[override] """ Checks if the list contains the specified item. Args: item (HT): The item to check for. Returns: bool: True if the item is in the list, False otherwise. """ return item in self._set @typing.overload def __setitem__(self, indices: typing.SupportsIndex, values: HT) -> None: """Overload: assign a single value at an integer index.""" @typing.overload def __setitem__( self, indices: slice, values: collections.abc.Iterable[HT] ) -> None: """Overload: assign an iterable of values to a slice.""" def __setitem__( self, indices: slice | typing.SupportsIndex, values: collections.abc.Iterable[HT] | HT, ) -> None: """ Sets the item(s) at the specified index/indices, ensuring uniqueness. Args: indices (slice | typing.SupportsIndex): The index or slice to set the value(s) at. values (Iterable[HT] | HT): The value(s) to set. Raises: RuntimeError: If `on_duplicate` is 'ignore' and setting slices. ValueError: If the value(s) are duplicates and `on_duplicate` is set to 'raise'. """ if isinstance(indices, slice): values = typing.cast(collections.abc.Iterable[HT], values) if self.on_duplicate == 'ignore': raise RuntimeError( 'ignore mode while setting slices introduces ambiguous ' 'behaviour and is therefore not supported' ) duplicates: set[HT] = set(values) & self._set if duplicates and values != list(self[indices]): raise ValueError(f'Duplicate values: {duplicates}') self._set.update(values) else: values = typing.cast(HT, values) if values in self._set and values != self[indices]: if self.on_duplicate == 'raise': raise ValueError(f'Duplicate value: {values}') else: return self._set.add(values) super().__setitem__( typing.cast(slice, indices), typing.cast(list[HT], values) ) def __delitem__(self, index: typing.SupportsIndex | slice) -> None: """ Deletes the item(s) at the specified index/indices. Args: index (typing.SupportsIndex | slice): The index or slice to delete the item(s) at. """ if isinstance(index, slice): for value in self[index]: self._set.remove(value) else: self._set.remove(self[index]) super().__delitem__(index) # Type hinting `collections.deque` does not work consistently between Python # runtime, mypy and pyright currently so we have to ignore the errors class SliceableDeque(typing.Generic[T], collections.deque[T]): """ A deque that supports slicing and enhanced equality checks. Methods:: __getitem__(index: typing.SupportsIndex | slice) -> T | 'SliceableDeque[T]': Returns the item or slice at the given index. __eq__(other: typing.Any) -> bool: Checks equality with another object, allowing for comparison with lists, tuples, and sets. pop(index: int = -1) -> T: Removes and returns the item at the given index. Only supports index 0 and the last index. """ @typing.overload def __getitem__(self, index: typing.SupportsIndex) -> T: """Overload: an integer index returns a single item.""" @typing.overload def __getitem__(self, index: slice) -> 'SliceableDeque[T]': """Overload: a slice returns a new ``SliceableDeque``.""" def __getitem__( self, index: typing.SupportsIndex | slice ) -> T | 'SliceableDeque[T]': """ Return the item or slice at the given index. Args: index (typing.SupportsIndex | slice): The index or slice to retrieve. Returns: T | 'SliceableDeque[T]': The item or slice at the given index. Examples: >>> d = SliceableDeque[int]([1, 2, 3, 4, 5]) >>> d[1:4] SliceableDeque([2, 3, 4]) >>> d = SliceableDeque[str](['a', 'b', 'c']) >>> d[-2:] SliceableDeque(['b', 'c']) """ if isinstance(index, slice): start, stop, step = index.indices(len(self)) return self.__class__(self[i] for i in range(start, stop, step)) else: return super().__getitem__(index) def __eq__(self, other: typing.Any) -> bool: """ Checks equality with another object, allowing for comparison with lists, tuples, and sets. Args: other (typing.Any): The object to compare with. Returns: bool: True if the objects are equal, False otherwise. """ if isinstance(other, list): return list(self) == other elif isinstance(other, tuple): return tuple(self) == other elif isinstance(other, set): return set(self) == other else: return super().__eq__(other) def pop(self, index: int = -1) -> T: """ Removes and returns the item at the given index. Only supports index 0 and the last index. Args: index (int, optional): The index of the item to remove. Defaults to -1. Returns: T: The removed item. Raises: IndexError: If the index is not 0 or the last index. Examples: >>> d = SliceableDeque([1, 2, 3]) >>> d.pop(0) 1 >>> d.pop() 3 """ if index == 0: return super().popleft() elif index in {-1, len(self) - 1}: return super().pop() else: raise IndexError( 'Only index 0 and the last index (`N-1` or `-1`) are supported' ) if __name__ == '__main__': import doctest doctest.testmod() python_utils-4.0.0/python_utils/converters.py0000644000000351450000000000000151600ustar """ This module provides utility functions for type conversion. Functions:: to_int: Convert a string to an integer with optional regexp matching. to_float: Convert a string to a float with optional regexp matching. to_unicode: Convert objects to Unicode strings. to_str: Convert objects to byte strings. scale_1024: Scale a number down to a suitable size (powers of 1024). remap: Remap a value from one range to another. """ # Ignoring all mypy errors because mypy doesn't understand many modern typing # constructs... please, use pyright instead if you can. from __future__ import annotations import decimal import math import re import typing from python_utils import _aliases #: Numeric type variable for ``remap`` (any ``int``, ``float`` or ``Decimal``). _TN = typing.TypeVar('_TN', bound=_aliases.DecimalNumber) #: Accepted ``regexp`` for ``to_int``/``to_float``: a compiled pattern, a #: pattern string, ``True`` for the built-in digit pattern, or ``None``. _RegexpType: typing.TypeAlias = ( re.Pattern[str] | str | typing.Literal[True] | None ) def to_int( input_: str | None = None, default: int = 0, exception: _aliases.ExceptionsType = (ValueError, TypeError), regexp: _RegexpType = None, ) -> int: r""" Convert the given input to an integer or return default. When trying to convert the exceptions given in the exception parameter are automatically caught and the default will be returned. The regexp parameter allows for a regular expression to find the digits in a string. When True it will automatically match any digit in the string. When a (regexp) object (has a search method) is given, that will be used. WHen a string is given, re.compile will be run over it first The last group of the regexp will be used as value >>> to_int('abc') 0 >>> to_int('1') 1 >>> to_int('') 0 >>> to_int() 0 >>> to_int('abc123') 0 >>> to_int('123abc') 0 >>> to_int('abc123', regexp=True) 123 >>> to_int('123abc', regexp=True) 123 >>> to_int('abc123abc', regexp=True) 123 >>> to_int('abc123abc456', regexp=True) 123 >>> to_int('abc123', regexp=re.compile(r'(\d+)')) 123 >>> to_int('123abc', regexp=re.compile(r'(\d+)')) 123 >>> to_int('abc123abc', regexp=re.compile(r'(\d+)')) 123 >>> to_int('abc123abc456', regexp=re.compile(r'(\d+)')) 123 >>> to_int('abc123', regexp=r'(\d+)') 123 >>> to_int('123abc', regexp=r'(\d+)') 123 >>> to_int('abc', regexp=r'(\d+)') 0 >>> to_int('abc123abc', regexp=r'(\d+)') 123 >>> to_int('abc123abc456', regexp=r'(\d+)') 123 >>> to_int('1234', default=1) 1234 >>> to_int('abc', default=1) 1 >>> to_int('abc', regexp=123) Traceback (most recent call last): ... TypeError: unknown argument for regexp parameter: 123 """ if regexp is True: regexp = re.compile(r'(\d+)') elif isinstance(regexp, str): regexp = re.compile(regexp) elif hasattr(regexp, 'search'): pass elif regexp is not None: raise TypeError(f'unknown argument for regexp parameter: {regexp!r}') try: if regexp and input_ and (match := regexp.search(input_)): input_ = match.groups()[-1] if input_ is None: return default else: return int(input_) except exception: return default def to_float( input_: str, default: int = 0, exception: _aliases.ExceptionsType = (ValueError, TypeError), regexp: _RegexpType = None, ) -> _aliases.Number: r""" Convert the given `input_` to an integer or return default. When trying to convert the exceptions given in the exception parameter are automatically caught and the default will be returned. The regexp parameter allows for a regular expression to find the digits in a string. When True it will automatically match any digit in the string. When a (regexp) object (has a search method) is given, that will be used. When a string is given, re.compile will be run over it first The last group of the regexp will be used as value >>> '%.2f' % to_float('abc') '0.00' >>> '%.2f' % to_float('1') '1.00' >>> '%.2f' % to_float('abc123.456', regexp=True) '123.46' >>> '%.2f' % to_float('abc123', regexp=True) '123.00' >>> '%.2f' % to_float('abc0.456', regexp=True) '0.46' >>> '%.2f' % to_float('abc123.456', regexp=re.compile(r'(\d+\.\d+)')) '123.46' >>> '%.2f' % to_float('123.456abc', regexp=re.compile(r'(\d+\.\d+)')) '123.46' >>> '%.2f' % to_float('abc123.46abc', regexp=re.compile(r'(\d+\.\d+)')) '123.46' >>> '%.2f' % to_float('abc123abc456', regexp=re.compile(r'(\d+(\.\d+|))')) '123.00' >>> '%.2f' % to_float('abc', regexp=r'(\d+)') '0.00' >>> '%.2f' % to_float('abc123', regexp=r'(\d+)') '123.00' >>> '%.2f' % to_float('123abc', regexp=r'(\d+)') '123.00' >>> '%.2f' % to_float('abc123abc', regexp=r'(\d+)') '123.00' >>> '%.2f' % to_float('abc123abc456', regexp=r'(\d+)') '123.00' >>> '%.2f' % to_float('1234', default=1) '1234.00' >>> '%.2f' % to_float('abc', default=1) '1.00' >>> '%.2f' % to_float('abc', regexp=123) Traceback (most recent call last): ... TypeError: unknown argument for regexp parameter """ if regexp is True: regexp = re.compile(r'(\d+(\.\d+|))') elif isinstance(regexp, str): regexp = re.compile(regexp) elif hasattr(regexp, 'search'): pass elif regexp is not None: raise TypeError('unknown argument for regexp parameter') try: if regexp and (match := regexp.search(input_)): input_ = match.group(1) return float(input_) except exception: return default def to_unicode( input_: _aliases.StringTypes, encoding: str = 'utf-8', errors: str = 'replace', ) -> str: """Convert objects to unicode, if needed decodes string with the given encoding and errors settings. :rtype: str >>> to_unicode(b'a') 'a' >>> to_unicode('a') 'a' >>> to_unicode('a') 'a' >>> class Foo(object): ... __str__ = lambda s: 'a' >>> to_unicode(Foo()) 'a' >>> to_unicode(Foo) "" """ if isinstance(input_, bytes): input_ = input_.decode(encoding, errors) else: input_ = str(input_) return input_ def to_str( input_: _aliases.StringTypes, encoding: str = 'utf-8', errors: str = 'replace', ) -> bytes: """Convert objects to string, encodes to the given encoding. :rtype: str >>> to_str('a') b'a' >>> to_str('a') b'a' >>> to_str(b'a') b'a' >>> class Foo(object): ... __str__ = lambda s: 'a' >>> to_str(Foo()) 'a' >>> to_str(Foo) "" """ if not isinstance(input_, bytes): if not hasattr(input_, 'encode'): input_ = str(input_) input_ = input_.encode(encoding, errors) return input_ def scale_1024( x: _aliases.Number, n_prefixes: int, ) -> tuple[_aliases.Number, _aliases.Number]: """Scale a number down to a suitable size, based on powers of 1024. Returns the scaled number and the power of 1024 used. Use to format numbers of bytes to KiB, MiB, etc. >>> scale_1024(310, 3) (310.0, 0) >>> scale_1024(2048, 3) (2.0, 1) >>> scale_1024(0, 2) (0.0, 0) >>> scale_1024(0.5, 2) (0.5, 0) >>> scale_1024(1, 2) (1.0, 0) """ if x <= 0: power = 0 else: power = min(int(math.log(x, 2) / 10), n_prefixes - 1) scaled = float(x) / (2 ** (10 * power)) return scaled, power @typing.overload def remap( value: decimal.Decimal, old_min: decimal.Decimal | float, old_max: decimal.Decimal | float, new_min: decimal.Decimal | float, new_max: decimal.Decimal | float, ) -> decimal.Decimal: """Overload: a ``Decimal`` ``value`` yields a ``Decimal`` result.""" @typing.overload def remap( value: decimal.Decimal | float, old_min: decimal.Decimal, old_max: decimal.Decimal | float, new_min: decimal.Decimal | float, new_max: decimal.Decimal | float, ) -> decimal.Decimal: """Overload: a ``Decimal`` ``old_min`` yields a ``Decimal`` result.""" @typing.overload def remap( value: decimal.Decimal | float, old_min: decimal.Decimal | float, old_max: decimal.Decimal, new_min: decimal.Decimal | float, new_max: decimal.Decimal | float, ) -> decimal.Decimal: """Overload: a ``Decimal`` ``old_max`` yields a ``Decimal`` result.""" @typing.overload def remap( value: decimal.Decimal | float, old_min: decimal.Decimal | float, old_max: decimal.Decimal | float, new_min: decimal.Decimal, new_max: decimal.Decimal | float, ) -> decimal.Decimal: """Overload: a ``Decimal`` ``new_min`` yields a ``Decimal`` result.""" @typing.overload def remap( value: decimal.Decimal | float, old_min: decimal.Decimal | float, old_max: decimal.Decimal | float, new_min: decimal.Decimal | float, new_max: decimal.Decimal, ) -> decimal.Decimal: """Overload: a ``Decimal`` ``new_max`` yields a ``Decimal`` result.""" # Note that float captures both int and float types so we don't need to # specify them separately @typing.overload def remap( value: float, old_min: float, old_max: float, new_min: float, new_max: float, ) -> float: """Overload: all-``float`` (or ``int``) inputs yield a ``float``.""" def remap( # pyright: ignore[reportInconsistentOverload] value: _TN, old_min: _TN, old_max: _TN, new_min: _TN, new_max: _TN, ) -> _TN: """ Remap a value from one range into another. >>> remap(500, 0, 1000, 0, 100) 50 >>> remap(250.0, 0.0, 1000.0, 0.0, 100.0) 25.0 >>> remap(-75, -100, 0, -1000, 0) -750 >>> remap(33, 0, 100, -500, 500) -170 >>> remap(decimal.Decimal('250.0'), 0.0, 1000.0, 0.0, 100.0) Decimal('25.0') This is a great use case example. Take an AVR that has dB values the minimum being -80dB and the maximum being 10dB and you want to convert volume percent to the equivalent in that dB range >>> remap(46.0, 0.0, 100.0, -80.0, 10.0) -38.6 I added using decimal.Decimal so floating point math errors can be avoided. Here is an example of a floating point math error >>> 0.1 + 0.1 + 0.1 0.30000000000000004 If floating point remaps need to be done my suggestion is to pass at least one parameter as a `decimal.Decimal`. This will ensure that the output from this function is accurate. I left passing `floats` for backwards compatibility and there is no conversion done from float to `decimal.Decimal` unless one of the passed parameters has a type of `decimal.Decimal`. This will ensure that any existing code that uses this function will work exactly how it has in the past. Some edge cases to test >>> remap(1, 0, 0, 1, 2) Traceback (most recent call last): ... ValueError: Input range (0-0) is empty >>> remap(1, 1, 2, 0, 0) Traceback (most recent call last): ... ValueError: Output range (0-0) is empty Args: value (int, float, decimal.Decimal): Value to be converted. old_min (int, float, decimal.Decimal): Minimum of the range for the value that has been passed. old_max (int, float, decimal.Decimal): Maximum of the range for the value that has been passed. new_min (int, float, decimal.Decimal): The minimum of the new range. new_max (int, float, decimal.Decimal): The maximum of the new range. Returns: int, float, decimal.Decimal: Value that has been re-ranged. If any of the parameters passed is a `decimal.Decimal`, all of the parameters will be converted to `decimal.Decimal`. The same thing also happens if one of the parameters is a `float`. Otherwise, all parameters will get converted into an `int`. Technically, you can pass a `str` of an integer and it will get converted. The returned value type will be `decimal.Decimal` if any of the passed parameters are `decimal.Decimal`, the return type will be `float` if any of the passed parameters are a `float`, otherwise the returned type will be `int`. """ # Promote every argument to one common type: Decimal if any input is a # Decimal, otherwise float if any is a float, otherwise int. This preserves # the caller's precision (see above) instead of silently downcasting. type_: type[_aliases.DecimalNumber] if ( isinstance(value, decimal.Decimal) or isinstance(old_min, decimal.Decimal) or isinstance(old_max, decimal.Decimal) or isinstance(new_min, decimal.Decimal) or isinstance(new_max, decimal.Decimal) ): type_ = decimal.Decimal elif ( isinstance(value, float) or isinstance(old_min, float) or isinstance(old_max, float) or isinstance(new_min, float) or isinstance(new_max, float) ): type_ = float else: type_ = int value = typing.cast(_TN, type_(value)) old_min = typing.cast(_TN, type_(old_min)) old_max = typing.cast(_TN, type_(old_max)) new_max = typing.cast(_TN, type_(new_max)) new_min = typing.cast(_TN, type_(new_min)) # These might not be floats but the Python type system doesn't understand # the generic type system in this case old_range = typing.cast(float, old_max) - typing.cast(float, old_min) new_range = typing.cast(float, new_max) - typing.cast(float, new_min) if old_range == 0: raise ValueError(f'Input range ({old_min}-{old_max}) is empty') if new_range == 0: raise ValueError(f'Output range ({new_min}-{new_max}) is empty') # The current state of Python typing makes it impossible to use the # generic type system in this case. Or so extremely verbose that it's not # worth it. new_value = (value - old_min) * new_range # type: ignore[operator] # pyright: ignore[reportOperatorIssue, reportUnknownVariableType] # Integer inputs use floor division to keep the result integral; float and # Decimal inputs use true division. if type_ is int: new_value //= old_range # pyright: ignore[reportUnknownVariableType] # pyrefly: ignore[unsupported-operation] else: new_value /= old_range # pyright: ignore[reportUnknownVariableType] # pyrefly: ignore[unsupported-operation] new_value += new_min # type: ignore[operator] # pyright: ignore[reportOperatorIssue, reportUnknownVariableType] return typing.cast(_TN, new_value) python_utils-4.0.0/python_utils/decorators.py0000644000000157240000000000000151340ustar """ This module provides various utility decorators for Python functions and methods. The decorators include: 1. `set_attributes`: Sets attributes on functions and classes. 2. `listify`: Converts any generator to a list or other collection. 3. `sample`: Limits calls to a function based on a sample rate. 4. `wraps_classmethod`: Wraps classmethods with type info from a regular method. Each decorator is designed to enhance the functionality of Python functions and methods in a simple and reusable manner. """ import collections.abc import contextlib import functools import logging import random import typing _T = typing.TypeVar('_T') _P = typing.ParamSpec('_P') def set_attributes( **kwargs: typing.Any, ) -> collections.abc.Callable[..., typing.Any]: """Decorator to set attributes on functions and classes. A common usage for this pattern is the Django Admin where functions can get an optional short_description. To illustrate: Example from the Django admin using this decorator: https://docs.djangoproject.com/en/3.0/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_display Our simplified version: >>> @set_attributes(short_description='Name') ... def upper_case_name(self, obj): ... return ('%s %s' % (obj.first_name, obj.last_name)).upper() The standard Django version: >>> def upper_case_name(obj): ... return ('%s %s' % (obj.first_name, obj.last_name)).upper() >>> upper_case_name.short_description = 'Name' """ def _set_attributes( function: collections.abc.Callable[_P, _T], ) -> collections.abc.Callable[_P, _T]: """Attach the captured ``kwargs`` as attributes on ``function``.""" for key, value in kwargs.items(): setattr(function, key, value) return function return _set_attributes def listify( collection: collections.abc.Callable[ [collections.abc.Iterable[_T]], collections.abc.Collection[_T] ] = list, allow_empty: bool = True, ) -> collections.abc.Callable[ [collections.abc.Callable[..., collections.abc.Iterable[_T] | None]], collections.abc.Callable[..., collections.abc.Collection[_T]], ]: """ Convert any generator to a list or other type of collection. >>> @listify() ... def generator(): ... yield 1 ... yield 2 ... yield 3 >>> generator() [1, 2, 3] >>> @listify() ... def empty_generator(): ... pass >>> empty_generator() [] >>> @listify(allow_empty=False) ... def empty_generator_not_allowed(): ... pass >>> empty_generator_not_allowed() # doctest: +ELLIPSIS Traceback (most recent call last): ... TypeError: ... `allow_empty` is `False` >>> @listify(collection=set) ... def set_generator(): ... yield 1 ... yield 1 ... yield 2 >>> set_generator() {1, 2} >>> @listify(collection=dict) ... def dict_generator(): ... yield 'a', 1 ... yield 'b', 2 >>> dict_generator() {'a': 1, 'b': 2} """ def _listify( function: collections.abc.Callable[ ..., collections.abc.Iterable[_T] | None ], ) -> collections.abc.Callable[..., collections.abc.Collection[_T]]: """Materialize ``function``'s result into ``collection``.""" def __listify( *args: typing.Any, **kwargs: typing.Any ) -> collections.abc.Collection[_T]: """Call ``function`` and gather its result into ``collection``.""" result: collections.abc.Iterable[_T] | None = function( *args, **kwargs ) if result is None: if allow_empty: return collection(iter(())) else: raise TypeError( f'{function} returned `None` and `allow_empty` ' 'is `False`' ) else: return collection(result) return __listify return _listify def sample( sample_rate: float, ) -> collections.abc.Callable[ [collections.abc.Callable[_P, _T]], collections.abc.Callable[_P, _T | None], ]: """ Limit calls to a function based on given sample rate. Number of calls to the function will be roughly equal to sample_rate percentage. Usage: >>> @sample(0.5) ... def demo_function(*args, **kwargs): ... return 1 Calls to *demo_function* will be limited to 50% approximately. """ def _sample( function: collections.abc.Callable[_P, _T], ) -> collections.abc.Callable[_P, _T | None]: """Wrap ``function`` so it only runs on a sampled fraction of calls.""" @functools.wraps(function) def __sample(*args: _P.args, **kwargs: _P.kwargs) -> _T | None: """Run ``function`` with probability ``sample_rate``, else skip.""" if random.random() < sample_rate: return function(*args, **kwargs) else: logging.debug( 'Skipped execution of %r(%r, %r) due to sampling', function, args, kwargs, ) return None return __sample return _sample def wraps_classmethod( wrapped: collections.abc.Callable[typing.Concatenate[typing.Any, _P], _T], ) -> collections.abc.Callable[ [ collections.abc.Callable[typing.Concatenate[typing.Any, _P], _T], ], collections.abc.Callable[typing.Concatenate[typing.Any, _P], _T], ]: """Like ``functools.wraps``, but for wrapping classmethods. Copies the wrapped method's metadata (name, docstring and annotations) onto the wrapper, so a classmethod wrapper carries the type information of the regular method it stands in for. Args: wrapped: The method whose metadata should be copied onto the wrapper. Returns: A decorator that updates its wrapper with ``wrapped``'s metadata. """ def _wraps_classmethod( wrapper: collections.abc.Callable[ typing.Concatenate[typing.Any, _P], _T ], ) -> collections.abc.Callable[typing.Concatenate[typing.Any, _P], _T]: """Copy ``wrapped``'s metadata onto ``wrapper`` and return it.""" # For some reason `functools.update_wrapper` fails on some test # runs but not while running actual code with contextlib.suppress(AttributeError): wrapper = functools.update_wrapper( wrapper, wrapped, assigned=tuple( a for a in functools.WRAPPER_ASSIGNMENTS if a != '__annotations__' ), ) if annotations := getattr(wrapped, '__annotations__', {}): # Drop `self`: the wrapper is a classmethod, so it takes no `self`. annotations.pop('self', None) wrapper.__annotations__ = annotations return wrapper return _wraps_classmethod python_utils-4.0.0/python_utils/exceptions.py0000644000000023160000000000000151410ustar """ This module provides utility functions for raising and reraising exceptions. Functions:: raise_exception(exception_class, *args, **kwargs): Returns a function that raises an exception of the given type with the given arguments. reraise(*args, **kwargs): Reraises the current exception. """ import collections.abc import typing def raise_exception( exception_class: type[Exception], *args: typing.Any, **kwargs: typing.Any, ) -> collections.abc.Callable[..., None]: """ Returns a function that raises an exception of the given type with the given arguments. >>> raise_exception(ValueError, 'spam')('eggs') Traceback (most recent call last): ... ValueError: spam """ def raise_(*args_: typing.Any, **kwargs_: typing.Any) -> typing.Any: """Raise ``exception_class`` with the captured args.""" raise exception_class(*args, **kwargs) return raise_ def reraise(*args: typing.Any, **kwargs: typing.Any) -> typing.Any: """ Reraises the current exception. This function seems useless, but it can be useful when you need to pass a callable to another function that raises an exception. """ raise python_utils-4.0.0/python_utils/formatters.py0000644000000125140000000000000151470ustar """ This module provides utility functions for formatting strings and dates. Functions:: camel_to_underscore: Convert camel case naming to underscore/snake case. apply_recursive: Apply a function to all keys in a scope recursively. timesince: Return a string representing 'time since', e.g. 3 days ago. """ # pyright: reportUnnecessaryIsInstance=false import collections.abc import datetime import typing from python_utils import _aliases def camel_to_underscore(name: str) -> str: """Convert camel case style naming to underscore/snake case style naming. If there are existing underscores they will be collapsed with the to-be-added underscores. Multiple consecutive capital letters will not be split except for the last one. >>> camel_to_underscore('SpamEggsAndBacon') 'spam_eggs_and_bacon' >>> camel_to_underscore('Spam_and_bacon') 'spam_and_bacon' >>> camel_to_underscore('Spam_And_Bacon') 'spam_and_bacon' >>> camel_to_underscore('__SpamAndBacon__') '__spam_and_bacon__' >>> camel_to_underscore('__SpamANDBacon__') '__spam_and_bacon__' """ output: list[str] = [] for i, c in enumerate(name): if i > 0: pc = name[i - 1] if c.isupper() and not pc.isupper() and pc != '_': # Uppercase and the previous character isn't upper/underscore? # Add the underscore output.append('_') elif i > 3 and not c.isupper(): # Will return the last 3 letters to check if we are changing # case previous = name[i - 3 : i] if previous.isalpha() and previous.isupper(): output.insert(len(output) - 1, '_') output.append(c.lower()) return ''.join(output) def apply_recursive( function: collections.abc.Callable[[str], str], data: _aliases.OptionalScope = None, **kwargs: typing.Any, ) -> _aliases.OptionalScope: """ Apply a function to all keys in a scope recursively. >>> apply_recursive(camel_to_underscore, {'SpamEggsAndBacon': 'spam'}) {'spam_eggs_and_bacon': 'spam'} >>> apply_recursive( ... camel_to_underscore, ... { ... 'SpamEggsAndBacon': { ... 'SpamEggsAndBacon': 'spam', ... } ... }, ... ) {'spam_eggs_and_bacon': {'spam_eggs_and_bacon': 'spam'}} >>> a = {'a_b_c': 123, 'def': {'DeF': 456}} >>> b = apply_recursive(camel_to_underscore, a) >>> b {'a_b_c': 123, 'def': {'de_f': 456}} >>> apply_recursive(camel_to_underscore, None) """ if data is None: return None elif isinstance(data, dict): return { function(key): apply_recursive(function, value, **kwargs) for key, value in data.items() } else: return data def timesince( dt: datetime.datetime | datetime.timedelta, default: str = 'just now', ) -> str: """ Returns string representing 'time since' e.g. 3 days ago, 5 hours ago etc. >>> now = datetime.datetime.now() >>> timesince(now) 'just now' >>> timesince(now - datetime.timedelta(seconds=1)) '1 second ago' >>> timesince(now - datetime.timedelta(seconds=2)) '2 seconds ago' >>> timesince(now - datetime.timedelta(seconds=60)) '1 minute ago' >>> timesince(now - datetime.timedelta(seconds=61)) '1 minute and 1 second ago' >>> timesince(now - datetime.timedelta(seconds=62)) '1 minute and 2 seconds ago' >>> timesince(now - datetime.timedelta(seconds=120)) '2 minutes ago' >>> timesince(now - datetime.timedelta(seconds=121)) '2 minutes and 1 second ago' >>> timesince(now - datetime.timedelta(seconds=122)) '2 minutes and 2 seconds ago' >>> timesince(now - datetime.timedelta(seconds=3599)) '59 minutes and 59 seconds ago' >>> timesince(now - datetime.timedelta(seconds=3600)) '1 hour ago' >>> timesince(now - datetime.timedelta(seconds=3601)) '1 hour and 1 second ago' >>> timesince(now - datetime.timedelta(seconds=3602)) '1 hour and 2 seconds ago' >>> timesince(now - datetime.timedelta(seconds=3660)) '1 hour and 1 minute ago' >>> timesince(now - datetime.timedelta(seconds=3661)) '1 hour and 1 minute ago' >>> timesince(now - datetime.timedelta(seconds=3720)) '1 hour and 2 minutes ago' >>> timesince(now - datetime.timedelta(seconds=3721)) '1 hour and 2 minutes ago' >>> timesince(datetime.timedelta(seconds=3721)) '1 hour and 2 minutes ago' """ if isinstance(dt, datetime.timedelta): diff = dt else: now = datetime.datetime.now() diff = abs(now - dt) periods = ( (diff.days / 365, 'year', 'years'), (diff.days % 365 / 30, 'month', 'months'), (diff.days % 30 / 7, 'week', 'weeks'), (diff.days % 7, 'day', 'days'), (diff.seconds / 3600, 'hour', 'hours'), (diff.seconds % 3600 / 60, 'minute', 'minutes'), (diff.seconds % 60, 'second', 'seconds'), ) output: list[str] = [] for period, singular, plural in periods: int_period = int(period) if int_period == 1: output.append(f'{int_period} {singular}') elif int_period: output.append(f'{int_period} {plural}') if output: return f'{" and ".join(output[:2])} ago' return default python_utils-4.0.0/python_utils/generators.py0000644000000074520000000000000151370ustar """ This module provides generator utilities for batching items from iterables and async iterables. Functions: abatcher(generator, batch_size=None, interval=None): Asyncio generator wrapper that returns items with a given batch size or interval (whichever is reached first). batcher(iterable, batch_size=10): Generator wrapper that returns items with a given batch size. """ import asyncio import collections.abc import time import typing import python_utils from python_utils import _aliases #: Element type of the iterables being batched. _T = typing.TypeVar('_T') async def abatcher( generator: collections.abc.AsyncGenerator[_T, None] | collections.abc.AsyncIterator[_T], batch_size: int | None = None, interval: _aliases.delta_type | None = None, ) -> collections.abc.AsyncGenerator[list[_T], None]: """ Asyncio generator wrapper that returns items with a given batch size or interval (whichever is reached first). Args: generator: The async generator or iterator to batch. batch_size (typing.Optional[int], optional): The number of items per batch. Defaults to None. interval (typing.Optional[_aliases.delta_type], optional): The time interval to wait before yielding a batch. Defaults to None. Yields: collections.abc.AsyncGenerator[list[_T], None]: A generator that yields batches of items. """ batch: list[_T] = [] assert batch_size or interval, 'Must specify either batch_size or interval' # If interval is specified, use it to determine when to yield the batch # Alternatively set a really long timeout to keep the code simpler if interval: interval_s = python_utils.delta_to_seconds(interval) else: # Set the timeout to 10 years interval_s = 60 * 60 * 24 * 365 * 10.0 next_yield: float = time.perf_counter() + interval_s done: set[asyncio.Task[_T]] pending: set[asyncio.Task[_T]] = set() while True: try: done, pending = await asyncio.wait( pending or [ asyncio.create_task( typing.cast( collections.abc.Coroutine[None, None, _T], generator.__anext__(), ) ), ], timeout=interval_s, return_when=asyncio.FIRST_COMPLETED, ) if done: batch.extend(result.result() for result in done) except StopAsyncIteration: if batch: yield batch break if batch_size is not None and len(batch) == batch_size: yield batch batch = [] if interval and batch and time.perf_counter() > next_yield: yield batch batch = [] # Always set the next yield time to the current time. If the # loop is running slow due to blocking functions we do not # want to burst too much next_yield = time.perf_counter() + interval_s def batcher( iterable: collections.abc.Iterable[_T], batch_size: int = 10, ) -> collections.abc.Generator[list[_T], None, None]: """ Generator wrapper that returns items with a given batch size. Args: iterable (collections.abc.Iterable[_T]): The iterable to batch. batch_size (int, optional): The number of items per batch. Defaults to 10. Yields: collections.abc.Generator[list[_T], None, None]: A generator that yields batches of items. """ batch: list[_T] = [] for item in iterable: batch.append(item) if len(batch) == batch_size: yield batch batch = [] if batch: yield batch python_utils-4.0.0/python_utils/import_.py0000644000000075140000000000000144360ustar """ This module provides utilities for importing modules and handling exceptions. Classes: DummyError(Exception): A custom exception class used as a default for exception handling. Functions: import_global(name, modules=None, exceptions=DummyError, locals_=None, globals_=None, level=-1): Imports the requested items into the global scope, with support for relative imports and custom exception handling. """ import typing from python_utils import _aliases class DummyError(Exception): """A custom exception class used as a default for exception handling.""" #: Backwards-compatible legacy alias for ``DummyError``. DummyException = DummyError def import_global( # noqa: C901 name: str, modules: list[str] | None = None, exceptions: _aliases.ExceptionsType = DummyError, locals_: _aliases.OptionalScope = None, globals_: _aliases.OptionalScope = None, level: int = -1, ) -> typing.Any: # sourcery skip: hoist-if-from-if """Import the requested items into the global scope. WARNING! this method _will_ overwrite your global scope If you have a variable named `path` and you call `import_global('sys')` it will be overwritten with `sys.path` Args: name (str): the name of the module to import, e.g. sys modules (str): the modules to import, use None for everything exceptions (Exception): the exception to catch, e.g. ImportError locals_: the `locals()` method (in case you need a different scope) globals_: the `globals()` method (in case you need a different scope) level (int): the level to import from, this can be used for relative imports """ frame = None name_parts: list[str] = name.split('.') modules_set: set[str] = set() try: # If locals_ or globals_ are not given, autodetect them by inspecting # the current stack if locals_ is None or globals_ is None: import inspect frame = inspect.stack()[1][0] if locals_ is None: locals_ = frame.f_locals if globals_ is None: globals_ = frame.f_globals try: # Relative imports are supported (from .spam import eggs) if not name_parts[0]: name_parts = name_parts[1:] level = 1 # raise IOError((name, level)) module = __import__( name=name_parts[0] or '.', globals=globals_, locals=locals_, fromlist=name_parts[1:], level=max(level, 0), ) # Make sure we get the right part of a dotted import (i.e. # spam.eggs should return eggs, not spam) try: for attr in name_parts[1:]: module = getattr(module, attr) except AttributeError as e: raise ImportError( 'No module named ' + '.'.join(name_parts) ) from e # If no list of modules is given, autodetect from either __all__ # or a dir() of the module if not modules: modules_set = set(getattr(module, '__all__', dir(module))) else: modules_set = set(modules).intersection(dir(module)) # Add all items in modules to the global scope for k in set(dir(module)).intersection(modules_set): if k and k[0] != '_': globals_[k] = getattr(module, k) except exceptions as e: return e finally: # Clean up, just to be sure del ( # pyrefly: ignore[unsupported-delete] name, name_parts, modules, modules_set, exceptions, locals_, globals_, frame, ) python_utils-4.0.0/python_utils/logger.py0000644000000265530000000000000142500ustar """ This module provides a base class `LoggerBase` and a derived class `Logged` for adding logging capabilities to classes. The `LoggerBase` class expects a `logger` attribute to be a `logging.Logger` or compatible instance and provides methods for logging at various levels. The `Logged` class automatically adds a named logger to the class. Classes: LoggerBase: A base class that adds logging utilities to a class. Logged: A derived class that automatically adds a named logger to a class. Example: >>> class MyClass(Logged): ... def __init__(self): ... Logged.__init__(self) >>> my_class = MyClass() >>> my_class.debug('debug') >>> my_class.info('info') >>> my_class.warning('warning') >>> my_class.error('error') >>> my_class.exception('exception') >>> my_class.log(0, 'log') """ from __future__ import annotations import abc import collections.abc import logging import types import typing if typing.TYPE_CHECKING: import typing_extensions from . import decorators __all__ = ['Logged'] #: Accepted values for the logging ``exc_info`` parameter: a bool, an #: ``(type, value, traceback)`` triple, a bare exception instance, or ``None``. # From the logging typeshed, converted to be compatible with Python 3.8 # https://github.com/python/typeshed/blob/main/stdlib/logging/__init__.pyi _ExcInfoType: typing.TypeAlias = ( bool | tuple[ type[BaseException], BaseException, types.TracebackType | None, ] | tuple[None, None, None] | BaseException | None ) #: Parameter specification capturing a wrapped logger method's arguments. _P = typing.ParamSpec('_P') #: Covariant return-type variable for wrapped logger methods. _T = typing.TypeVar('_T', covariant=True) class LoggerProtocol(typing.Protocol): """Structural type for any ``logging.Logger``-compatible logger. Objects that provide these methods (such as a ``logging.Logger`` or a ``loguru`` logger) can be used as the ``logger`` attribute of a ``LoggerBase`` subclass. Only the shape matters; no inheritance required. """ def debug( self, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``DEBUG`` level.""" def info( self, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``INFO`` level.""" def warning( self, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``WARNING`` level.""" def error( self, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``ERROR`` level.""" def critical( self, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``CRITICAL`` level.""" def exception( self, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``ERROR`` level, including exception information.""" def log( self, level: int, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at the integer ``level``.""" class LoggerBase(abc.ABC): """Class which automatically adds logging utilities to your class when inheriting. Expects `logger` to be a logging.Logger or compatible instance. Adds easy access to debug, info, warning, error, exception and log methods >>> class MyClass(LoggerBase): ... logger = logging.getLogger(__name__) ... ... def __init__(self): ... Logged.__init__(self) >>> my_class = MyClass() >>> my_class.debug('debug') >>> my_class.info('info') >>> my_class.warning('warning') >>> my_class.error('error') >>> my_class.exception('exception') >>> my_class.log(0, 'log') """ # I've tried using a protocol to properly type the logger but it gave all # sorts of issues with mypy so we're using the lazy solution for now. The # actual classes define the correct type anyway logger: typing.Any # logger: LoggerProtocol @classmethod def __get_name( # pyright: ignore[reportUnusedFunction] cls, *name_parts: str ) -> str: """Join the non-empty, stripped ``name_parts`` into a dotted name.""" return '.'.join(n.strip() for n in name_parts if n.strip()) @decorators.wraps_classmethod(logging.Logger.debug) @classmethod def debug( cls, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``DEBUG`` level on the class logger.""" return cls.logger.debug( # type: ignore[no-any-return] msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) @decorators.wraps_classmethod(logging.Logger.info) @classmethod def info( cls, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``INFO`` level on the class logger.""" return cls.logger.info( # type: ignore[no-any-return] msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) @decorators.wraps_classmethod(logging.Logger.warning) @classmethod def warning( cls, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``WARNING`` level on the class logger.""" return cls.logger.warning( # type: ignore[no-any-return] msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) @decorators.wraps_classmethod(logging.Logger.error) @classmethod def error( cls, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``ERROR`` level on the class logger.""" return cls.logger.error( # type: ignore[no-any-return] msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) @decorators.wraps_classmethod(logging.Logger.critical) @classmethod def critical( cls, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``CRITICAL`` level on the class logger.""" return cls.logger.critical( # type: ignore[no-any-return] msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) @decorators.wraps_classmethod(logging.Logger.exception) @classmethod def exception( cls, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at ``ERROR`` level with exception info attached.""" return cls.logger.exception( # type: ignore[no-any-return] msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) @decorators.wraps_classmethod(logging.Logger.log) @classmethod def log( cls, level: int, msg: object, *args: object, exc_info: _ExcInfoType = None, stack_info: bool = False, stacklevel: int = 1, extra: collections.abc.Mapping[str, object] | None = None, ) -> None: """Log ``msg`` at the integer ``level`` on the class logger.""" return cls.logger.log( # type: ignore[no-any-return] level, msg, *args, exc_info=exc_info, stack_info=stack_info, stacklevel=stacklevel, extra=extra, ) class Logged(LoggerBase): """Class which automatically adds a named logger to your class when inheriting. Adds easy access to debug, info, warning, error, exception and log methods >>> class MyClass(Logged): ... def __init__(self): ... Logged.__init__(self) >>> my_class = MyClass() >>> my_class.debug('debug') >>> my_class.info('info') >>> my_class.warning('warning') >>> my_class.error('error') >>> my_class.exception('exception') >>> my_class.log(0, 'log') >>> my_class._Logged__get_name('spam') 'spam' """ logger: logging.Logger # pragma: no cover @classmethod def __get_name(cls, *name_parts: str) -> str: """Build the dotted logger name via ``LoggerBase``'s joiner.""" return typing.cast( str, LoggerBase._LoggerBase__get_name(*name_parts), # type: ignore[attr-defined] # pyright: ignore[reportUnknownMemberType, reportUnknownVariableType, reportAttributeAccessIssue] ) def __new__( cls, *args: typing.Any, **kwargs: typing.Any ) -> typing_extensions.Self: """ Create a new instance of the class and initialize the logger. The logger is named using the module and class name. Args: *args: Variable length argument list. **kwargs: Arbitrary keyword arguments. Returns: An instance of the class. """ cls.logger = logging.getLogger( cls.__get_name(cls.__module__, cls.__name__) ) return super().__new__(cls) python_utils-4.0.0/python_utils/loguru.py0000644000000027040000000000000142760ustar """ This module provides a `Logurud` class that integrates the `loguru` logger with the base logging functionality defined in `logger_module.LoggerBase`. Classes: Logurud: A class that extends `LoggerBase` and uses `loguru` for logging. Usage example: >>> from python_utils.loguru import Logurud >>> class MyClass(Logurud): ... def __init__(self): ... Logurud.__init__(self) >>> my_class = MyClass() >>> my_class.logger.info('This is an info message') """ from __future__ import annotations import typing import loguru from . import logger as logger_module __all__ = ['Logurud'] class Logurud(logger_module.LoggerBase): """ A class that extends `LoggerBase` and uses `loguru` for logging. Attributes: logger (loguru.Logger): The `loguru` logger instance. """ logger: loguru.Logger def __new__(cls, *args: typing.Any, **kwargs: typing.Any) -> Logurud: """ Creates a new instance of `Logurud` and initializes the `loguru` logger. Args: *args (typing.Any): Variable length argument list. **kwargs (typing.Any): Arbitrary keyword arguments. Returns: Logurud: A new instance of `Logurud`. """ # `logger` is already declared at class scope; assign without # re-annotating to avoid an obscured-declaration error. cls.logger = loguru.logger.opt(depth=1) return super().__new__(cls) python_utils-4.0.0/python_utils/py.typed0000644000000000000000000000000140710ustar python_utils-4.0.0/python_utils/terminal.py0000644000000152470000000000000146020ustar """ This module provides functions to get the terminal size across different platforms. Functions: get_terminal_size: Get the current size of the terminal. _get_terminal_size_windows: Get terminal size on Windows. _get_terminal_size_tput: Get terminal size using `tput`. _get_terminal_size_linux: Get terminal size on Linux. Usage example: >>> width, height = get_terminal_size() """ from __future__ import annotations import contextlib import os import typing from . import converters #: A terminal size as ``(width, height)`` in character cells. Dimensions = tuple[int, int] #: A ``Dimensions`` tuple, or ``None`` if the size could not be determined. OptionalDimensions = Dimensions | None #: A raw terminal size as ``(width, height)`` strings, before ``int`` casting. _StrDimensions = tuple[str, str] #: A ``_StrDimensions`` tuple, or ``None`` if the size could not be read. _OptionalStrDimensions = _StrDimensions | None def get_terminal_size() -> Dimensions: # pragma: no cover """Get the current size of your terminal. Multiple returns are not always a good idea, but in this case it greatly simplifies the code so I believe it's justified. It's not the prettiest function but that's never really possible with cross-platform code. Returns: width, height: Two integers containing width and height """ w: int | None h: int | None with contextlib.suppress(Exception): # Default to 79 characters for IPython notebooks import IPython # type: ignore[import-not-found] ipython = IPython.get_ipython() # type: ignore[no-untyped-call] from ipykernel import zmqshell # type: ignore[import-not-found] if isinstance(ipython, zmqshell.ZMQInteractiveShell): return 79, 24 with contextlib.suppress(Exception): # This works for Python 3, but not Pypy3. Probably the best method if # it's supported so let's always try import shutil w, h = shutil.get_terminal_size() if w and h: # The off by one is needed due to progressbars in some cases, for # safety we'll always subtract it. return w - 1, h with contextlib.suppress(Exception): # Fall back to the COLUMNS/LINES environment variables when set. w = converters.to_int(os.environ.get('COLUMNS')) h = converters.to_int(os.environ.get('LINES')) if w and h: return w, h with contextlib.suppress(Exception): # Try the optional `blessings` library if it happens to be installed. import blessings # type: ignore[import-untyped] terminal = blessings.Terminal() w = terminal.width h = terminal.height if w and h: return w, h with contextlib.suppress(Exception): # The method can return None so we don't unpack it wh = _get_terminal_size_linux() if wh is not None and all(wh): return wh with contextlib.suppress(Exception): # Windows detection doesn't always work, let's try anyhow wh = _get_terminal_size_windows() if wh is not None and all(wh): return wh with contextlib.suppress(Exception): # needed for window's python in cygwin's xterm! wh = _get_terminal_size_tput() if wh is not None and all(wh): return wh return 79, 24 def _get_terminal_size_windows() -> OptionalDimensions: # pragma: no cover """Return the terminal size on Windows via the Win32 console API. Returns: The ``(width, height)`` in cells, or ``None`` if it cannot be read. """ res = None try: import ctypes # stdin handle is -10 # stdout handle is -11 # stderr handle is -12 h = ctypes.windll.kernel32.GetStdHandle(-12) # type: ignore[attr-defined] csbi = ctypes.create_string_buffer(22) res = ctypes.windll.kernel32.GetConsoleScreenBufferInfo(h, csbi) # type: ignore[attr-defined] except Exception: return None if res: import struct (_, _, _, _, _, left, top, right, bottom, _, _) = struct.unpack( 'hhhhHhhhhhh', csbi.raw ) w = right - left h = bottom - top return w, h else: return None def _get_terminal_size_tput() -> OptionalDimensions: # pragma: no cover """Return the terminal size by shelling out to ``tput``. This is mainly needed for Windows Python running under Cygwin's xterm. Returns: The ``(width, height)`` in cells, or ``None`` on any failure. """ # get terminal width src: http://stackoverflow.com/questions/263890/ try: import subprocess proc = subprocess.Popen( ['tput', 'cols'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) output = proc.communicate(input=None) w = int(output[0]) proc = subprocess.Popen( ['tput', 'lines'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, ) output = proc.communicate(input=None) h = int(output[0]) except Exception: return None else: return w, h def _get_terminal_size_linux() -> OptionalDimensions: # pragma: no cover """Return the terminal size on Unix-likes via ``ioctl`` or the environment. Tries ``TIOCGWINSZ`` on the standard file descriptors, then the controlling terminal, then the ``LINES``/``COLUMNS`` environment variables. Returns: The ``(width, height)`` in cells, or ``None`` if every method fails. """ def ioctl_gwinsz(fd: int) -> tuple[str, str] | None: """Query ``fd`` for its window size via the ``TIOCGWINSZ`` ioctl. Returns: The ``(rows, cols)`` window size, or ``None`` if the ioctl fails. """ try: import fcntl import struct import termios return typing.cast( _OptionalStrDimensions, struct.unpack( 'hh', fcntl.ioctl(fd, termios.TIOCGWINSZ, '1234'), # type: ignore[call-overload] ), ) except Exception: return None size: _OptionalStrDimensions size = ioctl_gwinsz(0) or ioctl_gwinsz(1) or ioctl_gwinsz(2) if not size: with contextlib.suppress(Exception): fd = os.open(os.ctermid(), os.O_RDONLY) size = ioctl_gwinsz(fd) os.close(fd) if not size: try: size = os.environ['LINES'], os.environ['COLUMNS'] except Exception: return None return int(size[1]), int(size[0]) python_utils-4.0.0/python_utils/time.py0000644000000352360000000000000137250ustar """ This module provides utility functions for handling time-related operations. Functions:: timedelta_to_seconds: Convert a timedelta to seconds (microseconds as fraction). delta_to_seconds: Convert a timedelta or numeric interval to seconds. delta_to_seconds_or_none: Convert a timedelta to seconds or return None. format_time: Format a timestamp (timedelta, datetime, or seconds). timeout_generator: Generate items from an iterable until a timeout. aio_timeout_generator: Async generate items from an iterable until a timeout. aio_generator_timeout_detector: Detect if an async generator has stalled. aio_generator_timeout_detector_decorator: Decorator for the detector. """ # pyright: reportUnnecessaryIsInstance=false import collections.abc import datetime import functools import itertools import time import typing import python_utils from python_utils import _aliases, exceptions #: Item type produced by the time/timeout generators. _T = typing.TypeVar('_T') #: Parameter specification for the timeout-detector decorator's target. _P = typing.ParamSpec('_P') #: The Unix epoch (1970-01-01) as a naive ``datetime``, used as a reference. # There might be a better way to get the epoch with tzinfo; please open a # pull request if you know one. epoch = datetime.datetime(year=1970, month=1, day=1) def timedelta_to_seconds(delta: datetime.timedelta) -> _aliases.Number: """Convert a timedelta to seconds with the microseconds as fraction. Note that this method has become largely obsolete with the `timedelta.total_seconds()` method introduced in Python 2.7. >>> from datetime import timedelta >>> '%d' % timedelta_to_seconds(timedelta(days=1)) '86400' >>> '%d' % timedelta_to_seconds(timedelta(seconds=1)) '1' >>> '%.6f' % timedelta_to_seconds(timedelta(seconds=1, microseconds=1)) '1.000001' >>> '%.6f' % timedelta_to_seconds(timedelta(microseconds=1)) '0.000001' """ # Only convert to float if needed if delta.microseconds: total = delta.microseconds * 1e-6 else: total = 0 total += delta.seconds total += delta.days * 60 * 60 * 24 return total def delta_to_seconds(interval: _aliases.delta_type) -> _aliases.Number: """ Convert a timedelta to seconds. >>> delta_to_seconds(datetime.timedelta(seconds=1)) 1 >>> delta_to_seconds(datetime.timedelta(seconds=1, microseconds=1)) 1.000001 >>> delta_to_seconds(1) 1 >>> delta_to_seconds('whatever') # doctest: +ELLIPSIS Traceback (most recent call last): ... TypeError: Unknown type ... """ if isinstance(interval, datetime.timedelta): return timedelta_to_seconds(interval) elif isinstance(interval, (int, float)): return interval else: raise TypeError(f'Unknown type {type(interval)}: {interval!r}') def delta_to_seconds_or_none( interval: _aliases.delta_type | None, ) -> _aliases.Number | None: """Convert a timedelta to seconds, passing ``None`` through unchanged. Args: interval: A timedelta or a number of seconds, or ``None``. Returns: The interval in seconds, or ``None`` when ``interval`` is ``None``. >>> delta_to_seconds_or_none(datetime.timedelta(seconds=2)) 2 >>> delta_to_seconds_or_none(None) is None True """ if interval is None: return None else: return delta_to_seconds(interval) def format_time( timestamp: _aliases.timestamp_type, precision: datetime.timedelta = datetime.timedelta(seconds=1), ) -> str: """Formats timedelta/datetime/seconds. >>> format_time('1') '0:00:01' >>> format_time(1.234) '0:00:01' >>> format_time(1) '0:00:01' >>> format_time(datetime.datetime(2000, 1, 2, 3, 4, 5, 6)) '2000-01-02 03:04:05' >>> format_time(datetime.date(2000, 1, 2)) '2000-01-02' >>> format_time(datetime.timedelta(seconds=3661)) '1:01:01' >>> format_time(None) '--:--:--' >>> format_time(format_time) # doctest: +ELLIPSIS Traceback (most recent call last): ... TypeError: Unknown type ... """ precision_seconds = precision.total_seconds() if isinstance(timestamp, str): timestamp = float(timestamp) if isinstance(timestamp, (int, float)): try: timestamp = datetime.timedelta(seconds=timestamp) except OverflowError: # pragma: no cover timestamp = None if isinstance(timestamp, datetime.timedelta): seconds = timestamp.total_seconds() # Truncate the number to the given precision seconds = seconds - (seconds % precision_seconds) return str(datetime.timedelta(seconds=seconds)) elif isinstance(timestamp, datetime.datetime): # pragma: no cover # Python 2 doesn't have the timestamp method if hasattr(timestamp, 'timestamp'): seconds = timestamp.timestamp() else: seconds = timedelta_to_seconds(timestamp - epoch) # Truncate the number to the given precision seconds = seconds - (seconds % precision_seconds) try: # pragma: no cover dt = datetime.datetime.fromtimestamp(seconds) except (ValueError, OSError): # pragma: no cover dt = datetime.datetime.max return str(dt) elif isinstance(timestamp, datetime.date): return str(timestamp) elif timestamp is None: return '--:--:--' else: raise TypeError(f'Unknown type {type(timestamp)}: {timestamp!r}') @typing.overload def _to_iterable( iterable: collections.abc.Callable[[], collections.abc.AsyncIterable[_T]] | collections.abc.AsyncIterable[_T], ) -> collections.abc.AsyncIterable[_T]: """Async overload: async iterable or factory in, async iterable out.""" @typing.overload def _to_iterable( iterable: collections.abc.Callable[[], collections.abc.Iterable[_T]] | collections.abc.Iterable[_T], ) -> collections.abc.Iterable[_T]: """Sync overload: sync iterable or factory in, sync iterable out.""" def _to_iterable( iterable: collections.abc.Iterable[_T] | collections.abc.Callable[[], collections.abc.Iterable[_T]] | collections.abc.AsyncIterable[_T] | collections.abc.Callable[[], collections.abc.AsyncIterable[_T]], ) -> collections.abc.Iterable[_T] | collections.abc.AsyncIterable[_T]: """Return ``iterable``, calling it first if it is a zero-arg callable.""" if callable(iterable): return iterable() else: return iterable def timeout_generator( timeout: _aliases.delta_type, interval: _aliases.delta_type = datetime.timedelta(seconds=1), iterable: collections.abc.Iterable[_T] | collections.abc.Callable[ [], collections.abc.Iterable[_T] ] = itertools.count, # type: ignore[assignment] interval_multiplier: float = 1.0, maximum_interval: _aliases.delta_type | None = None, ) -> collections.abc.Iterable[_T]: """ Generator that walks through the given iterable (a counter by default) until the float_timeout is reached with a configurable float_interval between items. This can be used to limit the time spent on a slow operation. This can be useful for testing slow APIs so you get a small sample of the data in a reasonable amount of time. >>> for i in timeout_generator(0.1, 0.06): ... # Put your slow code here ... print(i) 0 1 2 >>> timeout = datetime.timedelta(seconds=0.1) >>> interval = datetime.timedelta(seconds=0.06) >>> for i in timeout_generator(timeout, interval, itertools.count()): ... print(i) 0 1 2 >>> for i in timeout_generator(1, interval=0.1, iterable='ab'): ... print(i) a b >>> timeout = datetime.timedelta(seconds=0.1) >>> interval = datetime.timedelta(seconds=0.06) >>> for i in timeout_generator(timeout, interval, interval_multiplier=2): ... print(i) 0 1 2 """ float_interval: float = delta_to_seconds(interval) float_maximum_interval: float | None = delta_to_seconds_or_none( maximum_interval ) iterable_ = _to_iterable(iterable) end = delta_to_seconds(timeout) + time.perf_counter() for item in iterable_: yield item if time.perf_counter() >= end: break time.sleep(float_interval) float_interval *= interval_multiplier if float_maximum_interval: float_interval = min(float_interval, float_maximum_interval) async def aio_timeout_generator( timeout: _aliases.delta_type, # noqa: ASYNC109 interval: _aliases.delta_type = datetime.timedelta(seconds=1), iterable: collections.abc.AsyncIterable[_T] | collections.abc.Callable[..., collections.abc.AsyncIterable[_T]] | None = None, interval_multiplier: float = 1.0, maximum_interval: _aliases.delta_type | None = None, ) -> collections.abc.AsyncGenerator[_T, None]: """ Async generator that walks through the given async iterable (a counter by default) until the float_timeout is reached with a configurable float_interval between items. The interval_exponent automatically increases the float_timeout with each run. Note that if the float_interval is less than 1, 1/interval_exponent will be used so the float_interval is always growing. To double the float_interval with each run, specify 2. Doctests and asyncio are not friends, so no examples. But this function is effectively the same as the `timeout_generator` but it uses `async for` instead. """ # Imported lazily so that importing `python_utils.time` for its # synchronous helpers (e.g. ``format_time``) does not pull in ``asyncio``. import asyncio from python_utils import aio if iterable is None: iterable = typing.cast( collections.abc.Callable[[], collections.abc.AsyncIterable[_T]], aio.acount, ) float_interval: float = delta_to_seconds(interval) float_maximum_interval: float | None = delta_to_seconds_or_none( maximum_interval ) iterable_ = _to_iterable(iterable) end = delta_to_seconds(timeout) + time.perf_counter() async for item in iterable_: # pragma: no branch yield item if time.perf_counter() >= end: break await asyncio.sleep(float_interval) float_interval *= interval_multiplier if float_maximum_interval: # pragma: no branch float_interval = min(float_interval, float_maximum_interval) async def aio_generator_timeout_detector( generator: collections.abc.AsyncGenerator[_T, None], timeout: _aliases.delta_type | None = None, # noqa: ASYNC109 total_timeout: _aliases.delta_type | None = None, on_timeout: collections.abc.Callable[ [ collections.abc.AsyncGenerator[_T, None], _aliases.delta_type | None, _aliases.delta_type | None, BaseException, ], typing.Any, ] | None = exceptions.reraise, **on_timeout_kwargs: collections.abc.Mapping[str, typing.Any], ) -> collections.abc.AsyncGenerator[_T, None]: """ This function is used to detect if an asyncio generator has not yielded an element for a set amount of time. The `on_timeout` argument is called with the `generator`, `timeout`, `total_timeout`, `exception` and the extra `**kwargs` to this function as arguments. If `on_timeout` is not specified, the exception is reraised. If `on_timeout` is `None`, the exception is silently ignored and the generator will finish as normal. """ # Imported lazily so importing `python_utils.time` stays asyncio-free. import asyncio if total_timeout is None: total_timeout_end = None else: total_timeout_end = time.perf_counter() + delta_to_seconds( total_timeout ) timeout_s = python_utils.delta_to_seconds_or_none(timeout) while True: try: if total_timeout_end and time.perf_counter() >= total_timeout_end: raise asyncio.TimeoutError( # noqa: TRY301 'Total timeout reached' ) if timeout_s: yield await asyncio.wait_for(generator.__anext__(), timeout_s) else: yield await generator.__anext__() except asyncio.TimeoutError as exception: # noqa: PERF203 if on_timeout is not None: await on_timeout( generator, timeout, total_timeout, exception, **on_timeout_kwargs, ) break except StopAsyncIteration: break def aio_generator_timeout_detector_decorator( timeout: _aliases.delta_type | None = None, total_timeout: _aliases.delta_type | None = None, on_timeout: collections.abc.Callable[ [ collections.abc.AsyncGenerator[typing.Any, None], _aliases.delta_type | None, _aliases.delta_type | None, BaseException, ], typing.Any, ] | None = exceptions.reraise, **on_timeout_kwargs: collections.abc.Mapping[str, typing.Any], ) -> collections.abc.Callable[ [collections.abc.Callable[_P, collections.abc.AsyncGenerator[_T, None]]], collections.abc.Callable[_P, collections.abc.AsyncGenerator[_T, None]], ]: """Wrap a generator function with ``aio_generator_timeout_detector``. Args: timeout: Per-item timeout; if a single yield takes longer, ``on_timeout`` fires. ``None`` disables the per-item check. total_timeout: Overall timeout across the whole generator. on_timeout: Callback invoked on a timeout; defaults to re-raising. **on_timeout_kwargs: Extra keyword arguments passed to ``on_timeout``. Returns: A decorator that wraps an async-generator function so every call is guarded against stalls. """ def _timeout_detector_decorator( generator: collections.abc.Callable[ _P, collections.abc.AsyncGenerator[_T, None] ], ) -> collections.abc.Callable[ _P, collections.abc.AsyncGenerator[_T, None] ]: """Wrap ``generator`` so each call is timeout-guarded.""" @functools.wraps(generator) def wrapper( *args: _P.args, **kwargs: _P.kwargs, ) -> collections.abc.AsyncGenerator[_T, None]: """Forward the call to ``aio_generator_timeout_detector``.""" return aio_generator_timeout_detector( generator(*args, **kwargs), timeout, total_timeout, on_timeout, **on_timeout_kwargs, ) return wrapper return _timeout_detector_decorator python_utils-4.0.0/python_utils/types.py0000644000000104320000000000000141220ustar """ This module provides type definitions and utility functions for type hinting. It includes: - Shorthand for commonly used types such as Optional and Union. - Type aliases for various data structures and common types. - Importing all types from the `typing` and `typing_extensions` modules. - Importing specific types from the `types` module. The module also configures Pyright to ignore wildcard import warnings. """ # pyright: reportWildcardImportFromLibrary=false # ruff: noqa: F405 # Kept as public module attributes for backwards compatibility: # `from python_utils.types import datetime, decimal` worked in 3.x. import datetime as datetime import decimal as decimal from re import Match, Pattern from types import * # pragma: no cover # noqa: F403 from typing import * # pragma: no cover # noqa: F403 # `import *` does not import these on all Python versions. `O` / `U` are kept # as backwards-compatible shorthands; new code should prefer `X | None`. from typing import ( IO, BinaryIO, Optional as O, # noqa: N817 TextIO, Union as U, # noqa: N817 ) from typing_extensions import * # type: ignore[no-redef,assignment] # noqa: F403 # Lightweight aliases live in a stdlib-only module so importers don't pull in # typing_extensions; re-exported here to keep the public surface unchanged. # The redundant `X as X` form marks these as intentional re-exports so strict # type checkers and ruff see `python_utils.types.X` without adding them to # `__all__` (which would change `from python_utils.types import *`). from ._aliases import ( DecimalNumber as DecimalNumber, ExceptionType as ExceptionType, ExceptionsType as ExceptionsType, Number as Number, OptionalScope as OptionalScope, Scope as Scope, StringTypes as StringTypes, delta_type as delta_type, timestamp_type as timestamp_type, ) __all__ = [ 'IO', 'TYPE_CHECKING', # ABCs (from collections.abc). 'AbstractSet', # The types from the typing module. # Super-special typing primitives. 'Annotated', 'Any', # One-off things. 'AnyStr', 'AsyncContextManager', 'AsyncGenerator', 'AsyncGeneratorType', 'AsyncIterable', 'AsyncIterator', 'Awaitable', # Other concrete types. 'BinaryIO', 'BuiltinFunctionType', 'BuiltinMethodType', 'ByteString', 'Callable', # Concrete collection types. 'ChainMap', 'ClassMethodDescriptorType', 'ClassVar', 'CodeType', 'Collection', 'Concatenate', 'Container', 'ContextManager', 'Coroutine', 'CoroutineType', 'Counter', 'DecimalNumber', 'DefaultDict', 'Deque', 'Dict', 'DynamicClassAttribute', 'Final', 'ForwardRef', 'FrameType', 'FrozenSet', # Types from the `types` module. 'FunctionType', 'Generator', 'GeneratorType', 'Generic', 'GetSetDescriptorType', 'Hashable', 'ItemsView', 'Iterable', 'Iterator', 'KeysView', 'LambdaType', 'List', 'Literal', 'Mapping', 'MappingProxyType', 'MappingView', 'Match', 'MemberDescriptorType', 'MethodDescriptorType', 'MethodType', 'MethodWrapperType', 'ModuleType', 'MutableMapping', 'MutableSequence', 'MutableSet', 'NamedTuple', # Not really a type. 'NewType', 'NoReturn', 'Number', 'O', 'Optional', 'OptionalScope', 'OrderedDict', 'ParamSpec', 'ParamSpecArgs', 'ParamSpecKwargs', 'Pattern', 'Protocol', # Structural checks, a.k.a. protocols. 'Reversible', 'Sequence', 'Set', 'SimpleNamespace', 'Sized', 'SupportsAbs', 'SupportsBytes', 'SupportsComplex', 'SupportsFloat', 'SupportsIndex', 'SupportsIndex', 'SupportsInt', 'SupportsRound', 'Text', 'TextIO', 'TracebackType', 'TracebackType', 'Tuple', 'Type', 'TypeAlias', 'TypeGuard', 'TypeVar', 'TypedDict', # Not really a type. 'U', 'Union', 'ValuesView', 'WrapperDescriptorType', 'cast', 'coroutine', 'delta_type', 'final', 'get_args', 'get_origin', 'get_type_hints', 'is_typeddict', 'new_class', 'no_type_check', 'no_type_check_decorator', 'overload', 'prepare_class', 'resolve_bases', 'runtime_checkable', 'timestamp_type', ] python_utils-4.0.0/tox.ini0000644000000042110000000000000111540ustar [tox] requires = tox>=4 tox-uv>=1 # Don't fail a local `tox` run when an interpreter from the matrix (e.g. 3.14 or # PyPy) isn't installed; CI provides each interpreter via tox-gh-actions. skip_missing_interpreters = true env_list = py{310,311,312,313,314} pypy{310,311} lint mypy pyright pyrefly codespell docs coverage labels = test = py{310,311,312,313,314}, pypy{310,311} check = lint, mypy, pyright, pyrefly, codespell [gh-actions] python = 3.10: py310 3.11: py311 3.12: py312 3.13: py313 3.14: py314 pypy-3.10: pypy310 pypy-3.11: pypy311 [testenv] runner = uv-venv-lock-runner extras = tests # Only the `tests` extra is needed; exclude the default `dev` group (its mypy -> # ast-serialize dependency has no PyPy wheel and would fail to build). uv_sync_flags = --no-default-groups # No lockfile is committed at this tier (see .gitignore); resolve freshly instead # of requiring `uv.lock` via the default `--locked`. uv_sync_locked = false # CI sets COVERAGE_FILE to a per-matrix-cell name (e.g. .coverage.3.12) so each # cell's data file is unique and the `coverage` job can combine them. pass_env = COVERAGE_FILE # Per-cell coverage gate is disabled; the `coverage` env / CI combine job is # the single 100% gate across the whole matrix. commands = pytest --cov-fail-under=0 {posargs} [testenv:lint] runner = uv-venv-runner dependency_groups = dev commands = ruff check . ruff format --check . [testenv:mypy] runner = uv-venv-runner dependency_groups = dev commands = mypy [testenv:pyright] runner = uv-venv-runner dependency_groups = dev commands = basedpyright [testenv:pyrefly] runner = uv-venv-runner dependency_groups = dev commands = pyrefly check [testenv:codespell] runner = uv-venv-runner dependency_groups = dev commands = codespell [testenv:docs] runner = uv-venv-runner extras = docs commands = sphinx-build -W -b html docs docs/_build/html [testenv:coverage] runner = uv-venv-lock-runner extras = tests uv_sync_flags = --no-default-groups uv_sync_locked = false commands = coverage erase pytest --cov=python_utils --cov-report=term-missing --cov-fail-under=100