pax_global_header00006660000000000000000000000064152172735330014522gustar00rootroot0000000000000052 comment=3127c657091c7c2b01f5d232b5ba9046bfbc1a80 apache-datafusion-python-54.0.0/000077500000000000000000000000001521727353300165035ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.ai/000077500000000000000000000000001521727353300171525ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.ai/skills/000077500000000000000000000000001521727353300204535ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.ai/skills/audit-skill-md/000077500000000000000000000000001521727353300232735ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.ai/skills/audit-skill-md/SKILL.md000066400000000000000000000313751521727353300245040ustar00rootroot00000000000000 --- name: audit-skill-md description: Audit the user-facing skill at skills/datafusion_python/SKILL.md against the current public Python API. Find new APIs that should be documented, stale mentions of removed/renamed APIs, examples that drifted from current idiomatic style, and places that need a "requires datafusion-python NN or newer" note. Run after upstream syncs and before each release. argument-hint: [scope] (e.g., "session-context", "dataframe", "expr", "functions", "patterns", "pitfalls", "version-notes", "all") --- # Audit `skills/datafusion_python/SKILL.md` You are auditing the user-facing skill at [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) against the current state of the Python API. The skill is the source of truth for how AI coding assistants are taught to write `datafusion-python` code, so it must match what the project actually ships. This skill identifies gaps caused by upstream syncs, refactors, or renames, and (if asked) applies the edits directly to `SKILL.md`. The skill is most usefully run **after** the `check-upstream` step of an upstream sync (see `dev/release/upstream-sync.md`) — once any new APIs are exposed, this skill makes sure they get documented. ## What the skill covers The user-facing `SKILL.md` documents these public surfaces. This list is not exhaustive — if a new top-level area is added (e.g., a new `Catalog` API exposed at the package root), include it. | Surface | Module | Sections in SKILL.md | |---|---|---| | `SessionContext` | `python/datafusion/context.py` | "Data Loading" | | `DataFrame` | `python/datafusion/dataframe.py` | "DataFrame Operations Quick Reference", "Executing and Collecting Results", "Idiomatic Patterns" | | `Expr` | `python/datafusion/expr.py` | "Expression Building", "Common Pitfalls" | | `functions` | `python/datafusion/functions/__init__.py` | "Available Functions (Categorized)", scattered uses throughout | | `functions.spark` | `python/datafusion/functions/spark.py` | "Available Functions (Categorized)" → "Spark-Compatible Functions" subsection | | Top-level helpers (`col`, `lit`, `WindowFrame`, ...) | `python/datafusion/__init__.py` | "Import Conventions", "Core Abstractions" | ## Scope argument The user may specify a scope via `$ARGUMENTS` to limit the audit. If no scope is given or `all` is specified, audit every area. | Scope | Audit target | |---|---| | `session-context` | `SessionContext` methods and the "Data Loading" section | | `dataframe` | `DataFrame` methods and the operations / executing / patterns sections | | `expr` | `Expr` methods/operators and the "Expression Building" section | | `functions` | `functions/__init__.py` `__all__` and the "Available Functions (Categorized)" section | | `spark-functions` | `functions/spark.py` `__all__`, the "Spark-Compatible Functions" subsection, and the divergent-semantics table | | `patterns` | "Idiomatic Patterns" section — confirm patterns still match recommended style | | `pitfalls` | "Common Pitfalls" — confirm each pitfall still reproduces, drop ones fixed upstream | | `version-notes` | Cross-check version annotations (see below) | | `all` | Everything above | ## Inputs to read Before producing the report: 1. `skills/datafusion_python/SKILL.md` — the document being audited. 2. The relevant Python module(s) for the chosen scope. Public surface is the `__all__` list (where defined) plus `class` and `def` symbols not prefixed with `_`. 3. `Cargo.toml` (root) for the current `datafusion-python` version — read the `version` field under `[workspace.package]` (format `NN.0.0`). The major version always matches the upstream `datafusion` crate, so a single `datafusion-python` version expresses both. `python/datafusion/__init__.py`'s `__version__` is the same value exposed at runtime. 4. Recent commits touching the relevant module(s) for context on what changed since the last sync: ```bash git log --oneline -- python/datafusion/dataframe.py | head -20 ``` ## What to look for Walk through each scoped area and flag four kinds of issues. ### 1. New APIs not mentioned For each public symbol in the module's `__all__` (or each public class method), check whether it appears anywhere in `SKILL.md`. A symbol is "covered" if it shows up in: - A code block (the strongest signal — it's demonstrated). - The "Available Functions (Categorized)" list. - The SQL-to-DataFrame Reference table. **Decide whether each missing symbol deserves an entry.** Not every public symbol belongs in `SKILL.md` — the skill is curated for the patterns users hit daily, not exhaustive API reference. Use these heuristics: - **Add it** if it replaces or supersedes something already in the skill (e.g., a new operation that is the idiomatic alternative to a documented workaround). - **Add it** if it fits a category already present (a new aggregate function goes in the aggregate list; a new join type goes in the joining section). - **Add it** if it changes how a documented pattern should be written. - **Skip it** if it is genuinely niche / advanced / experimental. - **Skip it** if it is internal plumbing exposed for FFI but not user-facing. When you flag a missing symbol, include a one-line proposed insertion point (which section / which table row) so a reviewer can decide quickly. ### 2. Stale mentions For each function name, method name, or import shown in `SKILL.md`, verify it still exists in the current API: - Function names mentioned in prose or in the categorized list should appear in `python/datafusion/functions/__init__.py`'s `__all__`. - Spark function names mentioned in the "Spark-Compatible Functions" subsection should appear in `python/datafusion/functions/spark.py`'s `__all__`. Also confirm the divergent-semantics table still matches the current spark vs. main signatures. - Method calls in code blocks should resolve against the current class. - Imports (`from datafusion import ...`) should succeed against the current `__init__.py`. A quick way to check imports without running them: ```bash python -c "from datafusion import SessionContext, col, lit; from datafusion import functions as F; print('ok')" ``` For each stale mention, propose either: - a rename to the current name, or - removal if the API is gone with no replacement. ### 3. Examples that drifted from idiomatic style The skill teaches a Pythonic style: prefer plain strings to `col(...)` when a column reference is all you need; prefer raw Python values to `lit(...)` where auto-wrapping applies. Recent refactors (see the `make-pythonic` skill) keep moving more functions toward accepting native types. For each code example in `SKILL.md`, check: - Does it use `lit(value)` where a raw value would work? Comparison RHS, arithmetic with a column, etc. all auto-wrap. (Reserve `lit()` for the cases listed in pitfall #2.) - Does it use `col("name")` where a plain string would work? `select(...)`, `aggregate([keys], ...)`, `sort(...)`, `sort_by(...)` all accept plain name strings. - Do `functions.py` calls match the current pythonic signature for that function? If `make-pythonic` recently changed a signature (e.g., `repeat(string, n: Expr | int)`), the example should pass `3` rather than `lit(3)`. - Does any example use a deprecated or removed parameter name? For drift, propose the updated snippet. If the change is purely stylistic and the older form still works, mark the suggestion as **non-blocking**. ### 4. Missing or stale version notes When an API depends on a specific version, the skill should say so — otherwise an agent referencing the skill in an older project will write code that fails at import or at runtime. `datafusion-python` shares its major version number with the upstream `datafusion` crate (e.g., `datafusion-python 53.x` tracks upstream `datafusion 53`). Always express version requirements in terms of `datafusion-python` only — there is no need to call out upstream and package versions separately. Add a version note when: - A method or function shown in the skill was added in a specific release (e.g., a new `DataFrame` method that didn't exist before 53). - A breaking change altered behavior in a specific release (signature change, default-value change, new required argument). - A pitfall was fixed in a specific release. Either annotate the pitfall block with "fixed in datafusion-python NN, kept here for users on older versions" or remove it once the supported floor moves past that version. Format for version notes (inline, italicized): ```markdown *Requires datafusion-python 53 or newer.* ``` For each missing/stale version note, propose the exact line and where it belongs. ## How to discover changes since the last audit If the user supplies a previous version or commit SHA where the audit was last run, diff against it: ```bash # Public-API-relevant changes since SHA git log --oneline ..HEAD -- python/datafusion/ # Whose signatures actually moved git diff ..HEAD -- python/datafusion/functions.py | grep '^[+-]def ' ``` If no prior audit point is given, fall back to "since the last upstream sync" by inspecting commits that touch `Cargo.toml`'s `datafusion` pin: ```bash git log --oneline -- Cargo.toml | grep -i datafusion | head -5 ``` ## Output Format Produce a report grouped by scope. Each finding is one bullet with a proposed action, so a maintainer can review the list quickly and apply edits in order. ``` ## SKILL.md Audit (scope: ) Audited against: - skills/datafusion_python/SKILL.md @ - datafusion-python ### New APIs to cover - `DataFrame.foo()` — added in datafusion-python 53. Insert in "DataFrame Operations Quick Reference" under . Proposed snippet: ```python df.foo(...) ``` ### Stale mentions - "old_function_name" referenced in the categorized list (line N) — renamed to "new_function_name". Replace. ### Drifted examples - "Filtering" section, `df.filter(col("a") > lit(10))` — drop `lit(10)`, auto-wrap applies. (non-blocking) - "Aggregation" section, `df.aggregate([col("region")], ...)` — pass `"region"` as a plain string per "Projection" guidance. ### Version notes - `DataFrame.foo()` block needs *Requires datafusion-python 53 or newer.* - "Common Pitfalls" #N — fixed in datafusion-python 53; remove the pitfall and update the SQL-to-DataFrame row to no longer flag the workaround. ### No-change confirmed - `SessionContext` data-loading section — all entries match current API. ``` If asked to apply the changes, edit `skills/datafusion_python/SKILL.md` directly with `Edit` tool calls, one finding at a time, and re-run the relevant doctest sanity check at the end: ```bash pytest --doctest-modules python/datafusion -q ``` ## What NOT to flag - **Internal helpers / underscored names.** Private symbols are not part of the user-facing surface. - **Functions intentionally omitted.** Niche / advanced APIs (custom catalogs, raw FFI plumbing, low-level execution plan accessors) live in the API reference, not the skill. If an omission was deliberate and a comment / commit explains why, leave it out. - **Style nits inside explanatory prose.** The skill mixes example code and prose; only enforce the pythonic style on actual code blocks. - **Function-by-function coverage of every `functions.py` symbol.** The "Available Functions (Categorized)" list is curated by category, not exhaustive. Adding a single new aggregate to the aggregate list is enough — the user follows the pointer to the API reference for the rest. ## Coordination with other skills - Run `/check-upstream` first to expose any missing upstream APIs into the Python layer. Without that, this skill cannot recommend documenting something that is not yet exposed. - Run `/make-pythonic` before this skill if a Pythonic-signature pass is planned for a release — that way this skill can update examples to the final signature in one shot rather than churning them twice. - The order during an upstream sync (PR 3 of `dev/release/upstream-sync.md`) is therefore: `/check-upstream` → `/make-pythonic` (optional) → `/audit-skill-md`. apache-datafusion-python-54.0.0/.ai/skills/check-upstream/000077500000000000000000000000001521727353300233665ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.ai/skills/check-upstream/SKILL.md000066400000000000000000000556531521727353300246040ustar00rootroot00000000000000 --- name: check-upstream description: Check if upstream Apache DataFusion features (functions, DataFrame ops, SessionContext methods, FFI types) are exposed in this Python project. Use when adding missing functions, auditing API coverage, or ensuring parity with upstream. argument-hint: [area] (e.g., "scalar functions", "aggregate functions", "window functions", "dataframe", "session context", "ffi types", "all") --- # Check Upstream DataFusion Feature Coverage You are auditing the datafusion-python project to find features from the upstream Apache DataFusion Rust library that are **not yet exposed** in this Python binding project. Your goal is to identify gaps and, if asked, implement the missing bindings. **IMPORTANT: The Python API is the source of truth for coverage.** A function or method is considered "exposed" if it exists in the Python API (e.g., `python/datafusion/functions.py`), even if there is no corresponding entry in the Rust bindings. Many upstream functions are aliases of other functions — the Python layer can expose these aliases by calling a different underlying Rust binding. Do NOT report a function as missing if it appears in the Python `__all__` list and has a working implementation, regardless of whether a matching `#[pyfunction]` exists in Rust. **IMPORTANT: audit the total upstream surface, not the delta since the last pin.** Gaps accumulate across syncs. A patch-release bump with a "bug fixes only" changelog does not mean there is nothing to find — pre-existing gaps from earlier majors still need to be surfaced. Always run the full comparison. ## Compile-Signal Triggers If a recent upstream bump required *any* of the following while fixing compile errors in `crates/core/` or the FFI example, treat that as a **hard signal** that user-facing surface area grew and run this skill before considering the bump done. Each pattern corresponds to a class of gap that frequently shows up in the audit: | Signal during PR 1 compile fix | Likely gap to check | |---|---| | New `Expr::*` variant added to a non-exhaustive `match` (`HigherOrderFunction`, `Lambda`, `LambdaVariable`, …) | New lambda / higher-order scalar functions (`any_match`, `array_transform`, `list_transform`, …) | | New `ScalarValue::*` variant (`ListView`, `LargeListView`, …) | New scalar / array functions that consume or produce the type | | New required trait method on `ExecutionPlan` / `TableProvider` / `*UDFImpl` (`apply_expressions`, …) | Corresponding capability on the Python wrapper class | | Renamed or restructured struct field (e.g. `Cast.data_type` → `Cast.field: FieldRef`) | Any Python accessor / SKILL.md doc that read the old field | | Newly deprecated trait method with a `_with_args` / `_with_options` replacement | The `*_with_options` variant frequently warrants a separate Python entry point | PR 1 of `dev/release/upstream-sync.md` asks you to log these signals as they appear. When you run this skill, use that log as a checklist: every entry must either show up in the audit output or be explicitly skipped with a reason. ## Areas to Check The user may specify an area via `$ARGUMENTS`. If no area is specified or "all" is given, check all areas. ### 1. Scalar Functions **Upstream source of truth:** - Rust docs: https://docs.rs/datafusion/latest/datafusion/functions/index.html - User docs: https://datafusion.apache.org/user-guide/sql/scalar_functions.html **Where they are exposed in this project:** - Python API: `python/datafusion/functions.py` — each function wraps a call to `datafusion._internal.functions` - Rust bindings: `crates/core/src/functions.rs` — `#[pyfunction]` definitions registered via `init_module()` **Evaluated and not requiring separate Python exposure:** - `get_field_path` — already covered by `get_field(expr, *names)`, which takes a variadic field path and dispatches to the same underlying `functions::core::get_field` UDF as the upstream `get_field_path` helper. **How to check:** 1. Fetch the upstream scalar function documentation page 2. Compare against functions listed in `python/datafusion/functions.py` (check the `__all__` list and function definitions) 3. A function is covered if it exists in the Python API — it does NOT need a dedicated Rust `#[pyfunction]`. Many functions are aliases that reuse another function's Rust binding. 4. Check against the "evaluated and not requiring exposure" list before flagging as a gap 5. Only report functions that are missing from the Python `__all__` list / function definitions ### 2. Aggregate Functions **Upstream source of truth:** - Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_aggregate/index.html - User docs: https://datafusion.apache.org/user-guide/sql/aggregate_functions.html **Where they are exposed in this project:** - Python API: `python/datafusion/functions.py` (aggregate functions are mixed in with scalar functions) - Rust bindings: `crates/core/src/functions.rs` **Evaluated and not requiring separate Python exposure:** - `count_distinct` — covered by `count(expr, distinct=True)`. Both forms call `count_udaf` with `distinct: bool = true` and produce the same logical plan. - `sum_distinct` — covered by `sum(expr, distinct=True)`. - `avg_distinct` — covered by `avg(expr, distinct=True)`. **How to check:** 1. Fetch the upstream aggregate function documentation page 2. Compare against aggregate functions in `python/datafusion/functions.py` (check `__all__` list and function definitions) 3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding 4. Check against the "evaluated and not requiring exposure" list before flagging as a gap 5. Report only functions missing from the Python API ### 3. Window Functions **Upstream source of truth:** - Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_window/index.html - User docs: https://datafusion.apache.org/user-guide/sql/window_functions.html **Where they are exposed in this project:** - Python API: `python/datafusion/functions.py` (window functions like `rank`, `dense_rank`, `lag`, `lead`, etc.) - Rust bindings: `crates/core/src/functions.rs` **How to check:** 1. Fetch the upstream window function documentation page 2. Compare against window functions in `python/datafusion/functions.py` (check `__all__` list and function definitions) 3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding 4. Report only functions missing from the Python API ### 4. Table Functions **Upstream source of truth:** - Rust docs: https://docs.rs/datafusion/latest/datafusion/functions_table/index.html - User docs: https://datafusion.apache.org/user-guide/sql/table_functions.html (if available) **Where they are exposed in this project:** - Python API: `python/datafusion/functions.py` and `python/datafusion/user_defined.py` (TableFunction/udtf) - Rust bindings: `crates/core/src/functions.rs` and `crates/core/src/udtf.rs` **How to check:** 1. Fetch the upstream table function documentation 2. Compare against what's available in the Python API 3. A function is covered if it exists in the Python API, even if it aliases another function's Rust binding 4. Report only functions missing from the Python API ### 5. DataFrame Operations **Upstream source of truth:** - Rust docs: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html **Where they are exposed in this project:** - Python API: `python/datafusion/dataframe.py` — the `DataFrame` class - Rust bindings: `crates/core/src/dataframe.rs` — `PyDataFrame` with `#[pymethods]` **Evaluated and not requiring separate Python exposure:** - `show_limit` — already covered by `DataFrame.show()`, which provides the same functionality with a simpler API - `with_param_values` — already covered by the `param_values` argument on `SessionContext.sql()`, which accomplishes the same thing more robustly - `union_by_name_distinct` — already covered by `DataFrame.union_by_name(distinct=True)`, which provides a more Pythonic API **How to check:** 1. Fetch the upstream DataFrame documentation page listing all methods 2. Compare against methods in `python/datafusion/dataframe.py` — this is the source of truth for coverage 3. The Rust bindings (`crates/core/src/dataframe.rs`) may be consulted for context, but a method is covered if it exists in the Python API 4. Check against the "evaluated and not requiring exposure" list before flagging as a gap 5. Report only methods missing from the Python API ### 6. SessionContext Methods **Upstream source of truth:** - Rust docs: https://docs.rs/datafusion/latest/datafusion/execution/context/struct.SessionContext.html **Where they are exposed in this project:** - Python API: `python/datafusion/context.py` — the `SessionContext` class - Rust bindings: `crates/core/src/context.rs` — `PySessionContext` with `#[pymethods]` **How to check:** 1. Fetch the upstream SessionContext documentation page listing all methods 2. Compare against methods in `python/datafusion/context.py` — this is the source of truth for coverage 3. The Rust bindings (`crates/core/src/context.rs`) may be consulted for context, but a method is covered if it exists in the Python API 4. Report only methods missing from the Python API ### 7. FFI Types (datafusion-ffi) **Upstream source of truth:** - Crate source: https://github.com/apache/datafusion/tree/main/datafusion/ffi/src - Rust docs: https://docs.rs/datafusion-ffi/latest/datafusion_ffi/ **Where they are exposed in this project:** - Rust bindings: various files under `crates/core/src/` and `crates/util/src/` - FFI example: `examples/datafusion-ffi-example/src/` - Dependency declared in root `Cargo.toml` and `crates/core/Cargo.toml` **Discovering currently supported FFI types:** Grep for `use datafusion_ffi::` in `crates/core/src/` and `crates/util/src/` to find all FFI types currently imported and used. **Evaluated and not requiring direct Python exposure:** These upstream FFI types have been reviewed and do not need to be independently exposed to end users: - `FFI_ExecutionPlan` — already used indirectly through table providers; no need for direct exposure - `FFI_PhysicalExpr` / `FFI_PhysicalSortExpr` — internal physical planning types not expected to be needed by end users - `FFI_RecordBatchStream` — one level deeper than FFI_ExecutionPlan, used internally when execution plans stream results - `FFI_SessionRef` / `ForeignSession` — session sharing across FFI; Python manages sessions natively via SessionContext - `FFI_SessionConfig` — Python can configure sessions natively without FFI - `FFI_ConfigOptions` / `FFI_TableOptions` — internal configuration plumbing - `FFI_PlanProperties` / `FFI_Boundedness` / `FFI_EmissionType` — read from existing plans, not user-facing - `FFI_Partitioning` — supporting type for physical planning - Supporting/utility types (`FFI_Option`, `FFI_Result`, `WrappedSchema`, `WrappedArray`, `FFI_ColumnarValue`, `FFI_Volatility`, `FFI_InsertOp`, `FFI_AccumulatorArgs`, `FFI_Accumulator`, `FFI_GroupsAccumulator`, `FFI_EmitTo`, `FFI_AggregateOrderSensitivity`, `FFI_PartitionEvaluator`, `FFI_PartitionEvaluatorArgs`, `FFI_Range`, `FFI_SortOptions`, `FFI_Distribution`, `FFI_ExprProperties`, `FFI_SortProperties`, `FFI_Interval`, `FFI_TableProviderFilterPushDown`, `FFI_TableType`) — used as building blocks within the types above, not independently exposed **How to check:** 1. Discover currently supported types by grepping for `use datafusion_ffi::` in `crates/core/src/` and `crates/util/src/`, then compare against the upstream `datafusion-ffi` crate's `lib.rs` exports 2. If new FFI types appear upstream, evaluate whether they represent a user-facing capability 3. Check against the "evaluated and not requiring exposure" list before flagging as a gap 4. Report any genuinely new types that enable user-facing functionality 5. For each currently supported FFI type, verify the full pipeline is present using the checklist from "Adding a New FFI Type": - Rust PyO3 wrapper with `from_pycapsule()` method - Python Protocol type (e.g., `ScalarUDFExportable`) for FFI objects - Python wrapper class with full type hints on all public methods - ABC base class (if the type can be user-implemented) - Registered in Rust `init_module()` and Python `__init__.py` - FFI example in `examples/datafusion-ffi-example/` - Type appears in union type hints where accepted ### 8. Spark-Compatible Functions (`datafusion-spark` crate) **Upstream source of truth:** - Crate source: https://github.com/apache/datafusion/tree/main/datafusion/spark/src - Rust docs: https://docs.rs/datafusion-spark/latest/datafusion_spark/ **Where they are exposed in this project:** - Python API: `python/datafusion/functions/spark.py` — each function wraps a call to `datafusion._internal.functions.spark`; the public surface is the module's `__all__` list. - Rust bindings: `crates/core/src/spark_functions.rs` — `#[pyfunction]` definitions registered via `init_module()` and re-exported under `datafusion._internal.functions.spark`. **Coverage policy:** The spark namespace mirrors `pyspark.sql.functions` parameter names and shapes exactly so pyspark callers can paste code unchanged. Extras over pyspark are permitted as long as positional pyspark calls still work — for example, the spark `avg` / `try_sum` / `collect_list` / `collect_set` retain the `distinct`/`filter`/`order_by`/`null_treatment` kwargs from the main namespace while pyspark's single-positional form continues to work. **How to check:** 1. Fetch the upstream `datafusion-spark` function list from the crate source under `datafusion/spark/src/function/` (each subdirectory is a category: `string/`, `math/`, `datetime/`, etc.). The crate's `function.rs` collects all `ScalarUDF` factories. 2. Cross-reference against `pyspark.sql.functions` for the public-facing shape — pyspark is the contract this namespace is matching. 3. Compare against the functions listed in `python/datafusion/functions/spark.py`'s `__all__`. A function is covered if it exists in the Python `spark` namespace, even if it aliases another function's Rust binding. 4. Report functions that are missing from the Python spark namespace. ### 9. `__all__` Hygiene (functions.py and functions/spark.py) Independent of upstream parity, also flag public `def` symbols in `python/datafusion/functions.py` **and** `python/datafusion/functions/spark.py` that are missing from that file's `__all__`. These are functions a user can call but that do not show up in `from datafusion.functions import *`, in tab-completion against the namespace, or in generated API docs — typically an oversight rather than an intentional omission. **How to check:** 1. Grep for `^def ([a-z_][a-z0-9_]*)\(` in each file to enumerate every public function definition. 2. Read the `__all__` list at the top of the same file. 3. Report any function in (1) that is not in (2). Skip private helpers (names starting with `_`). A historical example: `instr` and `position` shipped as public `def`s but were absent from `__all__` until the gap was caught here. For each finding, propose adding the name to `__all__` in alphabetical position with the existing entries. ## Checking for Existing GitHub Issues After identifying missing APIs, search the open issues at https://github.com/apache/datafusion-python/issues for each gap to see if an issue already exists requesting that API be exposed. Search using the function or method name as the query. - If an existing issue is found, include a link to it in the report. Do NOT create a new issue. - If no existing issue is found, note that no issue exists yet. If the user asks to create issues for missing APIs, each issue should specify that Python test coverage is required as part of the implementation. ## Output Format For each area checked, produce a report like: ``` ## [Area Name] Coverage Report ### Currently Exposed (X functions/methods) - list of what's already available ### Missing from Upstream (Y functions/methods) - function_name — brief description of what it does (existing issue: #123) - function_name — brief description of what it does (no existing issue) ### Notes - Any relevant observations about partial implementations, naming differences, etc. ``` ## Implementation Pattern If the user asks you to implement missing features, follow these patterns: ### Adding a New Function (Scalar/Aggregate/Window) **Step 1: Rust binding** in `crates/core/src/functions.rs`: ```rust #[pyfunction] #[pyo3(signature = (arg1, arg2))] fn new_function_name(arg1: PyExpr, arg2: PyExpr) -> PyResult { Ok(datafusion::functions::module::expr_fn::new_function_name(arg1.expr, arg2.expr).into()) } ``` Then register in `init_module()`: ```rust m.add_wrapped(wrap_pyfunction!(new_function_name))?; ``` **Step 2: Python wrapper** in `python/datafusion/functions.py`: ```python def new_function_name(arg1: Expr, arg2: Expr) -> Expr: """Description of what the function does. Args: arg1: Description of first argument. arg2: Description of second argument. Returns: Description of return value. """ return Expr(f.new_function_name(arg1.expr, arg2.expr)) ``` Add to `__all__` list. ### Adding a New DataFrame Method **Step 1: Rust binding** in `crates/core/src/dataframe.rs`: ```rust #[pymethods] impl PyDataFrame { fn new_method(&self, py: Python, param: PyExpr) -> PyDataFusionResult { let df = self.df.as_ref().clone().new_method(param.into())?; Ok(Self::new(df)) } } ``` **Step 2: Python wrapper** in `python/datafusion/dataframe.py`: ```python def new_method(self, param: Expr) -> DataFrame: """Description of the method.""" return DataFrame(self.df.new_method(param.expr)) ``` ### Adding a New SessionContext Method **Step 1: Rust binding** in `crates/core/src/context.rs`: ```rust #[pymethods] impl PySessionContext { pub fn new_method(&self, py: Python, param: String) -> PyDataFusionResult { let df = wait_for_future(py, self.ctx.new_method(¶m))?; Ok(PyDataFrame::new(df)) } } ``` **Step 2: Python wrapper** in `python/datafusion/context.py`: ```python def new_method(self, param: str) -> DataFrame: """Description of the method.""" return DataFrame(self.ctx.new_method(param)) ``` ### Adding a New FFI Type FFI types require a full pipeline from C struct through to a typed Python wrapper. Each layer must be present. **Step 1: Rust PyO3 wrapper class** in a new or existing file under `crates/core/src/`: ```rust use datafusion_ffi::new_type::FFI_NewType; #[pyclass(from_py_object, frozen, name = "RawNewType", module = "datafusion.module_name", subclass)] pub struct PyNewType { pub inner: Arc, } #[pymethods] impl PyNewType { #[staticmethod] fn from_pycapsule(obj: &Bound<'_, PyAny>) -> PyDataFusionResult { let capsule = obj .getattr("__datafusion_new_type__")? .call0()? .downcast::()?; let ffi_ptr = unsafe { capsule.reference::() }; let provider: Arc = ffi_ptr.into(); Ok(Self { inner: provider }) } fn some_method(&self) -> PyResult<...> { // wrap inner trait method } } ``` Register in the appropriate `init_module()`: ```rust m.add_class::()?; ``` **Step 2: Python Protocol type** in the appropriate Python module (e.g., `python/datafusion/catalog.py`): ```python class NewTypeExportable(Protocol): """Type hint for objects providing a __datafusion_new_type__ PyCapsule.""" def __datafusion_new_type__(self) -> object: ... ``` **Step 3: Python wrapper class** in the same module: ```python class NewType: """Description of the type. This class wraps a DataFusion NewType, which can be created from a native Python implementation or imported from an FFI-compatible library. """ def __init__( self, new_type: df_internal.module_name.RawNewType | NewTypeExportable, ) -> None: if isinstance(new_type, df_internal.module_name.RawNewType): self._raw = new_type else: self._raw = df_internal.module_name.RawNewType.from_pycapsule(new_type) def some_method(self) -> ReturnType: """Description of the method.""" return self._raw.some_method() ``` **Step 4: ABC base class** (if users should be able to subclass and provide custom implementations in Python): ```python from abc import ABC, abstractmethod class NewTypeProvider(ABC): """Abstract base class for implementing a custom NewType in Python.""" @abstractmethod def some_method(self) -> ReturnType: """Description of the method.""" ... ``` **Step 5: Module exports** — add to the appropriate `__init__.py`: - Add the wrapper class (`NewType`) to `python/datafusion/__init__.py` - Add the ABC (`NewTypeProvider`) if applicable - Add the Protocol type (`NewTypeExportable`) if it should be public **Step 6: FFI example** — add an example implementation under `examples/datafusion-ffi-example/src/`: ```rust // examples/datafusion-ffi-example/src/new_type.rs use datafusion_ffi::new_type::FFI_NewType; // ... example showing how an external Rust library exposes this type via PyCapsule ``` **Checklist for each FFI type:** - [ ] Rust PyO3 wrapper with `from_pycapsule()` method - [ ] Python Protocol type (e.g., `NewTypeExportable`) for FFI objects - [ ] Python wrapper class with full type hints on all public methods - [ ] ABC base class (if the type can be user-implemented) - [ ] Registered in Rust `init_module()` and Python `__init__.py` - [ ] FFI example in `examples/datafusion-ffi-example/` - [ ] Type appears in union type hints where accepted (e.g., `Table | TableProviderExportable`) ## Important Notes - The upstream DataFusion version used by this project is specified in `crates/core/Cargo.toml` — check the `datafusion` dependency version to ensure you're comparing against the right upstream version. - Some upstream features may intentionally not be exposed (e.g., internal-only APIs). Use judgment about what's user-facing. - When fetching upstream docs, prefer the published docs.rs documentation as it matches the crate version. - Function aliases (e.g., `array_append` / `list_append`) should both be exposed if upstream supports them. - Check the `__all__` list in `functions.py` to see what's publicly exported vs just defined. apache-datafusion-python-54.0.0/.ai/skills/make-pythonic/000077500000000000000000000000001521727353300232235ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.ai/skills/make-pythonic/SKILL.md000066400000000000000000000550461521727353300244350ustar00rootroot00000000000000 --- name: make-pythonic description: Audit and improve datafusion-python functions to accept native Python types (int, float, str, bool) instead of requiring explicit lit() or col() wrapping. Analyzes function signatures, checks upstream Rust implementations for type constraints, and applies the appropriate coercion pattern. argument-hint: [scope] (e.g., "string functions", "datetime functions", "array functions", "math functions", "all", or a specific function name like "split_part") --- # Make Python API Functions More Pythonic You are improving the datafusion-python API to feel more natural to Python users. The goal is to allow functions to accept native Python types (int, float, str, bool, etc.) for arguments that are contextually always or typically literal values, instead of requiring users to manually wrap them in `lit()`. **Core principle:** A Python user should be able to write `split_part(col("a"), ",", 2)` instead of `split_part(col("a"), lit(","), lit(2))` when the arguments are contextually obvious literals. ## Scope: `functions` vs `functions.spark` Both `python/datafusion/functions/__init__.py` and `python/datafusion/functions/spark.py` are in scope. We want both to feel pythonic — accept native Python types where the argument is contextually a literal — but `functions.spark` carries an additional constraint: **every signature must remain compatible with `pyspark.sql.functions`**. Compatibility rules for the spark namespace: - **Parameter names must match pyspark exactly.** Pyspark callers pass by keyword (`spark.shiftleft(col=..., numBits=...)`), so renames break them. Do NOT rename a parameter just because it would be more pythonic in the main namespace. - **Positional order must match pyspark exactly.** Reordering breaks positional pyspark calls. - **Type unions may widen the input set, never narrow it.** Pyspark accepts `Column` or `str` (column name) for most args; we accept `Expr` already, and widening to `Expr | int` / `Expr | str` for literal-friendly arguments is on-brand because the int/str case is exactly what a pyspark caller would also try. Just verify the widened set is a superset of what pyspark accepts for that arg. - **Extra keyword arguments are allowed** as long as they default to `None` and pyspark's positional/keyword form still works (e.g. the spark `avg`/`try_sum`/`collect_list`/`collect_set` retain DataFusion's `distinct`/`filter`/`order_by`/`null_treatment` kwargs). Practical effect: in `functions.spark`, apply Categories A and (where pyspark exposes the same arg as a non-`Expr`) B normally, but cross-check each proposed signature against `pyspark.sql.functions` before landing it. When pyspark's own type hint is `Column | str` for a "column name" arg, prefer leaving the spark wrapper at `Expr` — Category C ("`Expr | str` meaning column name") is unusual in `functions.py` and should remain so in `functions.spark`. ## How to Identify Candidates The user may specify a scope via `$ARGUMENTS`. If no scope is given or "all" is specified, audit all functions in `python/datafusion/functions/__init__.py` **and** `python/datafusion/functions/spark.py`. When updating a spark-namespace function, apply the compatibility rules from "Scope" above on top of the standard analysis. For each function, determine if any parameter can accept native Python types by evaluating **two complementary signals**: ### Signal 1: Contextual Understanding Some arguments are contextually always or almost always literal values based on what the function does: | Context | Typical Arguments | Examples | |---------|------------------|----------| | **String position/count** | Character counts, indices, repetition counts | `left(str, n)`, `right(str, n)`, `repeat(str, n)`, `lpad(str, count, ...)` | | **Delimiters/separators** | Fixed separator characters | `split_part(str, delim, idx)`, `concat_ws(sep, ...)` | | **Search/replace patterns** | Literal search strings, replacements | `replace(str, from, to)`, `regexp_replace(str, pattern, replacement, flags)` | | **Date/time parts** | Part names from a fixed set | `date_part(part, date)`, `date_trunc(part, date)` | | **Rounding precision** | Decimal place counts | `round(val, places)`, `trunc(val, places)` | | **Fill characters** | Padding characters | `lpad(str, count, fill)`, `rpad(str, count, fill)` | ### Signal 2: Upstream Rust Implementation Check the Rust binding in `crates/core/src/functions.rs` and the upstream DataFusion function implementation to determine type constraints. The upstream source is cached locally at: ``` ~/.cargo/registry/src/index.crates.io-*/datafusion-functions-/src/ ``` Check the DataFusion version in `crates/core/Cargo.toml` to find the right directory. Key subdirectories: `string/`, `datetime/`, `math/`, `regex/`. For **aggregate functions**, the upstream source is in a separate crate: ``` ~/.cargo/registry/src/index.crates.io-*/datafusion-functions-aggregate-/src/ ``` There are five concrete techniques to check, in order of signal strength: #### Technique 1: Check `invoke_with_args()` for literal-only enforcement (strongest signal) Some functions pattern-match on `ColumnarValue::Scalar` in their `invoke_with_args()` method and **return an error** if the argument is a column/array. This means the argument **must** be a literal — passing a column expression will fail at runtime. Example from `date_trunc.rs`: ```rust let granularity_str = if let ColumnarValue::Scalar(ScalarValue::Utf8(Some(v))) = granularity { v.to_lowercase() } else { return exec_err!("Granularity of `date_trunc` must be non-null scalar Utf8"); }; ``` **If you find this pattern:** The argument is **Category B** — accept only the corresponding native Python type (e.g., `str`), not `Expr`. The function will error at runtime with a column expression anyway. #### Technique 1a: Check `accumulator()` for literal-only enforcement (aggregate functions) Technique 1 applies to scalar UDFs. Aggregate functions do not have `invoke_with_args()` — instead, they enforce literal-only arguments in their `accumulator()` (or `create_accumulator()`) method, which runs at planning time before any data is processed. Look for these patterns inside `accumulator()`: - `get_scalar_value(expr)` — evaluates the expression against an empty batch and errors if it's not a scalar - `validate_percentile_expr(expr)` — specific helper used by percentile functions - `downcast_ref::()` — checks that the physical expression is a literal constant Example from `approx_percentile_cont.rs`: ```rust fn accumulator(&self, args: AccumulatorArgs) -> Result { let percentile = validate_percentile_expr(&args.exprs[1], "APPROX_PERCENTILE_CONT")?; // ... } ``` Where `validate_percentile_expr` calls `get_scalar_value` and errors with `"must be a literal"`. Example from `string_agg.rs`: ```rust fn accumulator(&self, acc_args: AccumulatorArgs) -> Result> { let Some(lit) = acc_args.exprs[1].as_any().downcast_ref::() else { return not_impl_err!( "The second argument of the string_agg function must be a string literal" ); }; // ... } ``` **If you find this pattern:** The argument is **Category B** — accept only the corresponding native Python type, not `Expr`. The function will error at planning time with a non-literal expression. To discover which aggregate functions have literal-only arguments, search the upstream aggregate crate for `get_scalar_value`, `validate_percentile_expr`, and `downcast_ref::()` inside `accumulator()` methods. For example, you should expect to find `approx_percentile_cont` (percentile) and `string_agg` (delimiter) among the results. #### Technique 1b: Check `partition_evaluator()` for literal-only enforcement (window functions) Window functions do not have `invoke_with_args()` or `accumulator()`. Instead, they enforce literal-only arguments in their `partition_evaluator()` method, which constructs the evaluator that processes each partition. The upstream source is in a separate crate: ``` ~/.cargo/registry/src/index.crates.io-*/datafusion-functions-window-/src/ ``` Look for `get_scalar_value_from_args()` calls inside `partition_evaluator()`. This helper (defined in the window crate's `utils.rs`) calls `downcast_ref::()` and errors with `"There is only support Literal types for field at idx: {index} in Window Function"`. Example from `ntile.rs`: ```rust fn partition_evaluator( &self, partition_evaluator_args: PartitionEvaluatorArgs, ) -> Result> { let scalar_n = get_scalar_value_from_args(partition_evaluator_args.input_exprs(), 0)? .ok_or_else(|| { exec_datafusion_err!("NTILE requires a positive integer") })?; // ... } ``` **If you find this pattern:** The argument is **Category B** — accept only the corresponding native Python type, not `Expr`. The function will error at planning time with a non-literal expression. To discover which window functions have literal-only arguments, search the upstream window crate for `get_scalar_value_from_args` inside `partition_evaluator()` methods. For example, you should expect to find `ntile` (n) and `lead`/`lag` (offset, default_value) among the results. #### Technique 2: Check the `Signature` for data type constraints Each function defines a `Signature::coercible(...)` that specifies what data types each argument accepts, using `Coercion` entries. This tells you the expected **data type** even if it doesn't enforce literal-only. Example from `repeat.rs`: ```rust signature: Signature::coercible( vec![ Coercion::new_exact(TypeSignatureClass::Native(logical_string())), Coercion::new_implicit( TypeSignatureClass::Native(logical_int64()), vec![TypeSignatureClass::Integer], NativeType::Int64, ), ], Volatility::Immutable, ), ``` This tells you arg 2 (`n`) must be an integer type coerced to Int64. Use this to choose the correct Python type (e.g., `int` not `str` or `float`). Common mappings: | Rust Type Constraint | Python Type | |---------------------|-------------| | `logical_int64()` / `TypeSignatureClass::Integer` | `int` | | `logical_float64()` / `TypeSignatureClass::Numeric` | `int \| float` | | `logical_string()` / `TypeSignatureClass::String` | `str` | | `LogicalType::Boolean` | `bool` | **Important:** In Python's type system (PEP 484), `float` already accepts `int` values, so `int | float` is redundant and will fail the `ruff` linter (rule PYI041). Use `float` alone when the Rust side accepts a float/numeric type — Python users can still pass integer literals like `log(10, col("a"))` or `power(col("a"), 3)` without issue. Only use `int` when the Rust side strictly requires an integer (e.g., `logical_int64()`). #### Technique 3: Check `return_field_from_args()` for `scalar_arguments` usage Functions that inspect literal values at query planning time use `args.scalar_arguments.get(n)` in their `return_field_from_args()` method. This indicates the argument is **expected to be a literal** for optimal behavior (e.g., to determine output type precision), but may still work as a column. Example from `round.rs`: ```rust let decimal_places: Option = match args.scalar_arguments.get(1) { None => Some(0), Some(None) => None, // argument is not a literal (column) Some(Some(scalar)) if scalar.is_null() => Some(0), Some(Some(scalar)) => Some(decimal_places_from_scalar(scalar)?), }; ``` **If you find this pattern:** The argument is **Category A** — accept native types AND `Expr`. It works as a column but is primarily used as a literal. #### Decision flow ``` What kind of function is this? Scalar UDF: Is argument rejected at runtime if not a literal? (check invoke_with_args for ColumnarValue::Scalar-only match + exec_err!) → YES: Category B — accept only native type, no Expr → NO: continue below Aggregate: Is argument rejected at planning time if not a literal? (check accumulator() for get_scalar_value / validate_percentile_expr / downcast_ref::() + error) → YES: Category B — accept only native type, no Expr → NO: continue below Window: Is argument rejected at planning time if not a literal? (check partition_evaluator() for get_scalar_value_from_args / downcast_ref::() + error) → YES: Category B — accept only native type, no Expr → NO: continue below Does the Signature constrain it to a specific data type? → YES: Category A — accept Expr | → NO: Leave as Expr only ``` ## Coercion Categories When making a function more pythonic, apply the correct coercion pattern based on **what the argument represents**: ### Category A: Arguments That Should Accept Native Types AND Expr These are arguments that are *typically* literals but *could* be column references in advanced use cases. For these, accept a union type and coerce native types to `Expr.literal()`. **Type hint pattern:** `Expr | int`, `Expr | str`, `Expr | int | str`, etc. **When to use:** When the argument could plausibly come from a column in some use case (e.g., the repeat count might come from a column in a data-driven scenario). ```python def repeat(string: Expr, n: Expr | int) -> Expr: """Repeats the ``string`` to ``n`` times. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["ha"]}) >>> result = df.select( ... dfn.functions.repeat(dfn.col("a"), 3).alias("r")) >>> result.collect_column("r")[0].as_py() 'hahaha' """ if not isinstance(n, Expr): n = Expr.literal(n) return Expr(f.repeat(string.expr, n.expr)) ``` ### Category B: Arguments That Should ONLY Accept Specific Native Types These are arguments where an `Expr` never makes sense because the value must be a fixed literal known at query-planning time (not a per-row value). For these, accept only the native type(s) and wrap internally. **Type hint pattern:** `str`, `int`, `list[str]`, etc. (no `Expr` in the union) **When to use:** When the argument is from a fixed enumeration or is always a compile-time constant, **AND** the parameter was not previously typed as `Expr`: - Separator in `concat_ws` (already typed as `str` in the Rust binding) - Index in `array_position` (already typed as `int` in the Rust binding) - Values that the Rust implementation already accepts as native types **Backward compatibility rule:** If a parameter was previously typed as `Expr`, you **must** keep `Expr` in the union even if the Rust side requires a literal. Removing `Expr` would break existing user code like `date_part(lit("year"), col("a"))`. Use **Category A** instead — accept `Expr | str` — and let users who pass column expressions discover the runtime error from the Rust side. Never silently break backward compatibility. ```python def concat_ws(separator: str, *args: Expr) -> Expr: """Concatenates the list ``args`` with the separator. ``separator`` is already typed as ``str`` in the Rust binding, so there is no backward-compatibility concern. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"], "b": ["world"]}) >>> result = df.select( ... dfn.functions.concat_ws("-", dfn.col("a"), dfn.col("b")).alias("c")) >>> result.collect_column("c")[0].as_py() 'hello-world' """ args = [arg.expr for arg in args] return Expr(f.concat_ws(separator, args)) ``` ### Category C: Arguments That Should Accept str as Column Name In some contexts a string argument naturally refers to a column name rather than a literal. This is the pattern used by DataFrame methods. **Type hint pattern:** `Expr | str` **When to use:** Only when the string contextually means a column name (rare in `functions.py`, more common in DataFrame methods). ```python # Use _to_raw_expr() from expr.py for this pattern from datafusion.expr import _to_raw_expr def some_function(column: Expr | str) -> Expr: raw = _to_raw_expr(column) # str -> col(str) return Expr(f.some_function(raw)) ``` **IMPORTANT:** In `functions.py`, string arguments almost never mean column names. Functions operate on expressions, and column references should use `col()`. Category C applies mainly to DataFrame methods and context APIs, not to scalar/aggregate/window functions. Do NOT convert string arguments to column expressions in `functions.py` unless there is a very clear reason to do so. ## Implementation Steps For each function being updated: ### Step 1: Analyze the Function 1. Read the current Python function signature in `python/datafusion/functions/__init__.py` 2. Read the Rust binding in `crates/core/src/functions.rs` 3. Optionally check the upstream DataFusion docs for the function 4. Determine which category (A, B, or C) applies to each parameter ### Step 2: Update the Python Function 1. **Change the type hints** to accept native types (e.g., `Expr` -> `Expr | int`) 2. **Add coercion logic** at the top of the function body 3. **Update the docstring** examples to use the simpler calling convention 4. **Preserve backward compatibility** — existing code using `Expr` must still work ### Step 3: Update Alias Type Hints After updating a primary function, find all alias functions that delegate to it (e.g., `instr` and `position` delegate to `strpos`). Update each alias's **parameter type hints** to match the primary function's new signature. Do not add coercion logic to aliases — the primary function handles that. ### Step 4: Update Docstring Examples (primary functions only) Per the project's CLAUDE.md rules: - Every function must have doctest-style examples - Optional parameters need examples both without and with the optional args, using keyword argument syntax - Reuse the same input data across examples where possible **Update examples to demonstrate the pythonic calling convention:** ```python # BEFORE (old style - still works but verbose) dfn.functions.left(dfn.col("a"), dfn.lit(3)) # AFTER (new style - shown in examples) dfn.functions.left(dfn.col("a"), 3) ``` ### Step 5: Run Tests After making changes, run the doctests to verify: ```bash python -m pytest --doctest-modules python/datafusion/functions/__init__.py -v ``` ## Coercion Helper Pattern Use the coercion helpers from `datafusion.expr` to convert native Python values to `Expr`. These are the complement of `ensure_expr()` — where `ensure_expr` *rejects* non-`Expr` values, the coercion helpers *wrap* them via `Expr.literal()`. **For required parameters** use `coerce_to_expr`: ```python from datafusion.expr import coerce_to_expr def left(string: Expr, n: Expr | int) -> Expr: n = coerce_to_expr(n) return Expr(f.left(string.expr, n.expr)) ``` **For optional nullable parameters** use `coerce_to_expr_or_none`: ```python from datafusion.expr import coerce_to_expr, coerce_to_expr_or_none def regexp_count( string: Expr, pattern: Expr | str, start: Expr | int | None = None, flags: Expr | str | None = None, ) -> Expr: pattern = coerce_to_expr(pattern) start = coerce_to_expr_or_none(start) flags = coerce_to_expr_or_none(flags) return Expr( f.regexp_count( string.expr, pattern.expr, start.expr if start is not None else None, flags.expr if flags is not None else None, ) ) ``` Both helpers are defined in `python/datafusion/expr.py` alongside `ensure_expr`. Import them in `functions.py` via: ```python from datafusion.expr import coerce_to_expr, coerce_to_expr_or_none ``` ## What NOT to Change - **Do not change arguments that represent data columns.** If an argument is the primary data being operated on (e.g., the `string` in `left(string, n)` or the `array` in `array_sort(array)`), it should remain `Expr` only. Users should use `col()` for column references. - **Do not change variadic `*args: Expr` parameters.** These represent multiple expressions and should stay as `Expr`. - **Do not change arguments where the coercion is ambiguous.** If it is unclear whether a string should be a column name or a literal, leave it as `Expr` and let the user be explicit. - **Do not add coercion logic to simple aliases.** If a function is just `return other_function(...)`, the primary function handles coercion. However, you **must update the alias's type hints** to match the primary function's signature so that type checkers and documentation accurately reflect what the alias accepts. - **Do not change the Rust bindings.** All coercion happens in the Python layer. The Rust functions continue to accept `PyExpr`. ## Priority Order When auditing functions, process them in this order: 1. **Date/time functions** — `date_part`, `date_trunc`, `date_bin` — these have the clearest literal arguments 2. **String functions** — `left`, `right`, `repeat`, `lpad`, `rpad`, `split_part`, `substring`, `replace`, `regexp_replace`, `regexp_match`, `regexp_count` — common and verbose without coercion 3. **Math functions** — `round`, `trunc`, `power` — numeric literal arguments 4. **Array functions** — `array_slice`, `array_position`, `array_remove_n`, `array_replace_n`, `array_resize`, `array_element` — index and count arguments 5. **Other functions** — any remaining functions with literal arguments ## Output Format For each function analyzed, report: ``` ## [Function Name] **Current signature:** `function(arg1: Expr, arg2: Expr) -> Expr` **Proposed signature:** `function(arg1: Expr, arg2: Expr | int) -> Expr` **Category:** A (accepts native + Expr) **Arguments changed:** - `arg2`: Expr -> Expr | int (always a literal count) **Rust binding:** Takes PyExpr, wraps to literal internally **Status:** [Changed / Skipped / Needs Discussion] ``` If asked to implement (not just audit), make the changes directly and show a summary of what was updated. apache-datafusion-python-54.0.0/.asf.yaml000066400000000000000000000027171521727353300202250ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. notifications: commits: commits@datafusion.apache.org issues: github@datafusion.apache.org pullrequests: github@datafusion.apache.org jira_options: link label worklog github: description: "Apache DataFusion Python Bindings" homepage: https://datafusion.apache.org/python enabled_merge_buttons: squash: true merge: false rebase: false features: issues: true protected_branches: main: required_status_checks: # require branches to be up-to-date before merging strict: true # don't require any jobs to pass contexts: [] staging: whoami: asf-staging subdir: python publish: whoami: asf-site subdir: python apache-datafusion-python-54.0.0/.claude/000077500000000000000000000000001521727353300200165ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.claude/skills000077700000000000000000000000001521727353300233432../.ai/skillsustar00rootroot00000000000000apache-datafusion-python-54.0.0/.dockerignore000066400000000000000000000001231521727353300211530ustar00rootroot00000000000000.cargo .github .pytest_cache ci conda dev docs examples parquet target testing venvapache-datafusion-python-54.0.0/.github/000077500000000000000000000000001521727353300200435ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.github/ISSUE_TEMPLATE/000077500000000000000000000000001521727353300222265ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.github/ISSUE_TEMPLATE/bug_report.md000066400000000000000000000006021521727353300247160ustar00rootroot00000000000000--- name: Bug report about: Create a report to help us improve title: '' labels: bug assignees: '' --- **Describe the bug** A clear and concise description of what the bug is. **To Reproduce** Steps to reproduce the behavior: **Expected behavior** A clear and concise description of what you expected to happen. **Additional context** Add any other context about the problem here. apache-datafusion-python-54.0.0/.github/ISSUE_TEMPLATE/feature_request.md000066400000000000000000000013701521727353300257540ustar00rootroot00000000000000--- name: Feature request about: Suggest an idea for this project title: '' labels: enhancement assignees: '' --- **Is your feature request related to a problem or challenge? Please describe what you are trying to do.** A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*) **Describe the solution you'd like** A clear and concise description of what you want to happen. **Describe alternatives you've considered** A clear and concise description of any alternative solutions or features you've considered. **Additional context** Add any other context or screenshots about the feature request here. apache-datafusion-python-54.0.0/.github/actions/000077500000000000000000000000001521727353300215035ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.github/actions/build-wheel/000077500000000000000000000000001521727353300237045ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.github/actions/build-wheel/action.yml000066400000000000000000000106751521727353300257150ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # Composite action that builds a datafusion-python wheel with maturin. # Centralises the abi3-vs-free-threaded argument logic so platform jobs # stay short and changes to wheel-build flags happen in one place. name: "Build wheel" description: "Build datafusion-python wheel with maturin (abi3 or free-threaded)" inputs: target: description: "Rust target triple (e.g. x86_64-unknown-linux-gnu). Required when manylinux is set; ignored for native builds." required: false default: "" python-tag: description: "abi3 (covers 3.10..3.14 GIL builds) or a free-threaded interpreter such as 3.13t / 3.14t" required: true build-mode: description: "release or debug" required: true features: description: "Comma-separated extra features (in addition to those implied by the python-tag)" required: false default: "substrait" manylinux: description: "manylinux tag for maturin-action (e.g. 2_28). Leave empty to use uv-run maturin natively." required: false default: "" out-dir: description: "Output directory for built wheels" required: false default: "dist" outputs: args: description: "Computed maturin args (for debugging)" value: ${{ steps.args.outputs.args }} runs: using: "composite" steps: - name: Compute maturin args id: args shell: bash run: | set -euo pipefail FEATURES="${{ inputs.features }}" TAG="${{ inputs.python-tag }}" if [ "$TAG" = "abi3" ]; then # Default features include the `abi3` cargo feature. # One wheel covers Python 3.10..3.14 (GIL builds only). BUILD_ARGS="--features ${FEATURES}" else # Free-threaded build: disable abi3, force mimalloc back in, pin interpreter. if [ "${RUNNER_OS:-}" = "Windows" ]; then # Windows free-threaded builds ship as `python.exe` (no `tN` # suffix). Resolve sys.executable so the path is independent of # PATH ordering, and assert the interpreter is actually # free-threaded before we hand the wheel off. INTERP=$(python -c 'import sys; print(sys.executable)') python -c "import sysconfig, sys; \ v = sysconfig.get_config_var('Py_GIL_DISABLED'); \ sys.exit(0 if v == 1 else f'expected free-threaded interpreter, got Py_GIL_DISABLED={v!r} at {sys.executable}')" # Backslashes in BUILD_ARGS would be parsed as escapes when the # output is re-expanded in the next step; use forward slashes # (maturin/Rust accept them on Windows). INTERP="${INTERP//\\//}" else INTERP="python${TAG}" fi BUILD_ARGS="--no-default-features --features mimalloc,${FEATURES} --interpreter ${INTERP}" fi if [ "${{ inputs.build-mode }}" = "release" ]; then BUILD_ARGS="--release --strip ${BUILD_ARGS}" fi BUILD_ARGS="${BUILD_ARGS} --out ${{ inputs.out-dir }}" echo "args=${BUILD_ARGS}" >> "$GITHUB_OUTPUT" echo "maturin args: ${BUILD_ARGS}" - name: Build via maturin-action (manylinux container) if: inputs.manylinux != '' uses: PyO3/maturin-action@v1 with: target: ${{ inputs.target }} manylinux: ${{ inputs.manylinux }} maturin-version: "1.13.3" args: ${{ steps.args.outputs.args }} rustup-components: rust-std - name: Build via native maturin if: inputs.manylinux == '' shell: bash # Use `uvx` so maturin is available even when `uv sync` was skipped # (free-threaded matrix entries don't pre-populate the project venv). run: uvx maturin@1.13.3 build ${{ steps.args.outputs.args }} apache-datafusion-python-54.0.0/.github/dependabot.yml000066400000000000000000000022021521727353300226670ustar00rootroot00000000000000# # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # version: 2 updates: - package-ecosystem: "cargo" directory: "/" schedule: interval: "weekly" day: "saturday" open-pull-requests-limit: 20 target-branch: main - package-ecosystem: "github-actions" directory: "/" schedule: interval: "weekly" day: "sunday" open-pull-requests-limit: 20 target-branch: main apache-datafusion-python-54.0.0/.github/pull_request_template.md000066400000000000000000000020371521727353300250060ustar00rootroot00000000000000# Which issue does this PR close? Closes #. # Rationale for this change # What changes are included in this PR? # Are there any user-facing changes? apache-datafusion-python-54.0.0/.github/workflows/000077500000000000000000000000001521727353300221005ustar00rootroot00000000000000apache-datafusion-python-54.0.0/.github/workflows/build.yml000066400000000000000000000432551521727353300237330ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # Reusable workflow for running building # This ensures the same tests run for both debug (PRs) and release (main/tags) builds name: Build on: workflow_call: inputs: build_mode: description: 'Build mode: debug or release' required: true type: string run_wheels: description: 'Whether to build distribution wheels' required: false type: boolean default: false env: CARGO_TERM_COLOR: always RUST_BACKTRACE: 1 UV_LOCKED: true jobs: # ============================================ # Linting Jobs # ============================================ lint-rust: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - name: Setup Rust uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 with: toolchain: "nightly" components: rustfmt - name: Cache Cargo uses: Swatinem/rust-cache@v2 - name: Check formatting run: cargo +nightly fmt --all -- --check lint-python: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - name: Install Python uses: actions/setup-python@v5 with: python-version: "3.12" - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Install dependencies run: uv sync --dev --no-install-package datafusion - name: Run Ruff run: | uv run --no-project ruff check --output-format=github python/ uv run --no-project ruff format --check python/ - name: Run codespell run: | uv run --no-project codespell --toml pyproject.toml lint-toml: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - name: Install taplo uses: taiki-e/install-action@v2 with: tool: taplo-cli # if you encounter an error, try running 'taplo format' to fix the formatting automatically. - name: Check Cargo.toml formatting run: taplo format --check check-crates-patch: if: inputs.build_mode == 'release' && startsWith(github.ref, 'refs/tags/') runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - name: Ensure [patch.crates-io] is empty run: python3 dev/check_crates_patch.py generate-license: runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Install cargo-license uses: taiki-e/install-action@v2 with: tool: cargo-license - name: Generate license file run: uv run --no-project python ./dev/create_license.py - uses: actions/upload-artifact@v7 with: name: python-wheel-license path: LICENSE.txt # ============================================ # Build - Linux x86_64 # ============================================ build-manylinux-x86_64: needs: [generate-license, lint-rust, lint-python] name: Linux x86_64 (${{ matrix.python-tag }}) runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - run: rm LICENSE.txt - name: Download LICENSE.txt uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - name: Setup Rust uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Add extra swap for release build if: inputs.build_mode == 'release' run: | set -euxo pipefail sudo swapoff -a || true sudo rm -f /swapfile sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile free -h swapon --show - name: Build wheel uses: ./.github/actions/build-wheel with: target: x86_64-unknown-linux-gnu python-tag: ${{ matrix.python-tag }} build-mode: ${{ inputs.build_mode }} features: "protoc,substrait" manylinux: "2_28" # FFI test wheel only needs to be built once per platform; gate to abi3. - name: Build FFI test library if: matrix.python-tag == 'abi3' uses: PyO3/maturin-action@v1 with: target: x86_64-unknown-linux-gnu manylinux: "2_28" working-directory: examples/datafusion-ffi-example args: --out dist rustup-components: rust-std - name: Archive wheels uses: actions/upload-artifact@v7 with: name: dist-manylinux-x86_64-${{ matrix.python-tag }} path: dist/* - name: Archive FFI test wheel if: matrix.python-tag == 'abi3' uses: actions/upload-artifact@v7 with: name: test-ffi-manylinux-x86_64 path: examples/datafusion-ffi-example/dist/* # ============================================ # Build - Linux ARM64 # ============================================ build-manylinux-aarch64: needs: [generate-license, lint-rust, lint-python] name: Linux arm64 (${{ matrix.python-tag }}) runs-on: ubuntu-24.04-arm strategy: fail-fast: false matrix: python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - run: rm LICENSE.txt - name: Download LICENSE.txt uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - name: Setup Rust uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Add extra swap for release build if: inputs.build_mode == 'release' run: | set -euxo pipefail sudo swapoff -a || true sudo rm -f /swapfile sudo fallocate -l 8G /swapfile || sudo dd if=/dev/zero of=/swapfile bs=1M count=8192 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile free -h swapon --show - name: Build wheel uses: ./.github/actions/build-wheel with: target: aarch64-unknown-linux-gnu python-tag: ${{ matrix.python-tag }} build-mode: ${{ inputs.build_mode }} features: "protoc,substrait" manylinux: "2_28" - name: Archive wheels uses: actions/upload-artifact@v7 if: inputs.build_mode == 'release' with: name: dist-manylinux-aarch64-${{ matrix.python-tag }} path: dist/* # ============================================ # Build - macOS arm64 / Windows # ============================================ build-python-mac-win: needs: [generate-license, lint-rust, lint-python] name: ${{ matrix.os == 'macos-latest' && 'macOS arm64' || 'Windows x86_64' }} (${{ matrix.python-tag }}) runs-on: ${{ matrix.os }} strategy: fail-fast: false matrix: os: [macos-latest, windows-latest] python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 - run: rm LICENSE.txt - name: Download LICENSE.txt uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - name: Setup Python (free-threaded) if: matrix.python-tag != 'abi3' uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-tag }} freethreaded: true - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Install Protoc uses: arduino/setup-protoc@v3 with: version: "27.4" repo-token: ${{ secrets.GITHUB_TOKEN }} - name: Install dependencies if: matrix.python-tag == 'abi3' run: uv sync --dev --no-install-package datafusion # Clippy is interpreter-agnostic; run once per OS (against the abi3 entry) # so the matrix doesn't pay the cost three times. - name: Run Clippy if: matrix.os != 'windows-latest' && matrix.python-tag == 'abi3' run: cargo clippy --no-deps --all-targets --features substrait -- -D warnings - name: Build wheel uses: ./.github/actions/build-wheel with: python-tag: ${{ matrix.python-tag }} build-mode: ${{ inputs.build_mode }} features: "substrait" out-dir: "target/wheels" - name: List Windows wheels if: matrix.os == 'windows-latest' run: dir target\wheels\ # since the runner is dynamic shellcheck (from actionlint) can't infer this is powershell # so we specify it explicitly shell: powershell - name: List Mac wheels if: matrix.os != 'windows-latest' run: find target/wheels/ - name: Archive wheels uses: actions/upload-artifact@v7 if: inputs.build_mode == 'release' with: name: dist-${{ matrix.os }}-${{ matrix.python-tag }} path: target/wheels/* # ============================================ # Build - macOS x86_64 (release only) # ============================================ build-macos-x86_64: if: inputs.build_mode == 'release' needs: [generate-license, lint-rust, lint-python] name: macOS x86_64 (${{ matrix.python-tag }}) runs-on: macos-15-intel strategy: fail-fast: false matrix: python-tag: ["abi3", "3.14t"] steps: - uses: actions/checkout@v6 - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 - run: rm LICENSE.txt - name: Download LICENSE.txt uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - name: Cache Cargo uses: Swatinem/rust-cache@v2 with: key: ${{ inputs.build_mode }}-${{ matrix.python-tag }} - name: Setup Python (free-threaded) if: matrix.python-tag != 'abi3' uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-tag }} freethreaded: true - uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Install Protoc uses: arduino/setup-protoc@v3 with: version: "27.4" repo-token: ${{ secrets.GITHUB_TOKEN }} - name: Install dependencies if: matrix.python-tag == 'abi3' run: uv sync --dev --no-install-package datafusion - name: Build wheel uses: ./.github/actions/build-wheel with: python-tag: ${{ matrix.python-tag }} build-mode: ${{ inputs.build_mode }} features: "substrait" out-dir: "target/wheels" - name: List Mac wheels run: find target/wheels/ - name: Archive wheels uses: actions/upload-artifact@v7 with: name: dist-macos-aarch64-${{ matrix.python-tag }} path: target/wheels/* # ============================================ # Build - Source Distribution # ============================================ build-sdist: needs: [generate-license] name: Source distribution if: inputs.build_mode == 'release' runs-on: ubuntu-latest steps: - uses: actions/checkout@v6 - run: rm LICENSE.txt - name: Download LICENSE.txt uses: actions/download-artifact@v8 with: name: python-wheel-license path: . - run: cat LICENSE.txt - name: Build sdist uses: PyO3/maturin-action@v1 with: rust-toolchain: stable manylinux: auto rustup-components: rust-std rustfmt args: --release --sdist --out dist --features protoc,substrait - name: Assert sdist build does not generate wheels run: | if [ "$(ls -A target/wheels)" ]; then echo "Error: Sdist build generated wheels" exit 1 else echo "Directory is clean" fi shell: bash # ============================================ # Build - Source Distribution # ============================================ merge-build-artifacts: runs-on: ubuntu-latest name: Merge build artifacts if: inputs.build_mode == 'release' needs: - build-python-mac-win - build-macos-x86_64 - build-manylinux-x86_64 - build-manylinux-aarch64 - build-sdist steps: - name: Merge Build Artifacts uses: actions/upload-artifact/merge@v7 with: name: dist pattern: dist-* # ============================================ # Build - Documentation # ============================================ # Documentation build job that runs after wheels are built build-docs: name: Build docs runs-on: ubuntu-latest needs: [build-manylinux-x86_64] # Only need the Linux wheel for docs # Only run docs on main branch pushes, tags, or PRs if: github.event_name == 'push' || github.event_name == 'pull_request' steps: - name: Set target branch if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref_type == 'tag') id: target-branch run: | set -x if test '${{ github.ref }}' = 'refs/heads/main'; then echo "value=asf-staging" >> "$GITHUB_OUTPUT" elif test '${{ github.ref_type }}' = 'tag'; then echo "value=asf-site" >> "$GITHUB_OUTPUT" else echo "Unsupported input: ${{ github.ref }} / ${{ github.ref_type }}" exit 1 fi - name: Checkout docs sources uses: actions/checkout@v6 - name: Checkout docs target branch if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref_type == 'tag') uses: actions/checkout@v6 with: fetch-depth: 0 ref: ${{ steps.target-branch.outputs.value }} path: docs-target - name: Setup Python uses: actions/setup-python@v6 with: python-version: "3.10" - name: Install dependencies uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true # Download the Linux wheel built in the previous job. # Docs only need the abi3 wheel — interpreter doesn't matter for sphinx. - name: Download pre-built Linux wheel uses: actions/download-artifact@v8 with: name: dist-manylinux-x86_64-abi3 path: wheels/ # Install from the pre-built wheels - name: Install from pre-built wheels run: | set -x uv venv # Install documentation dependencies uv sync --dev --no-install-package datafusion --group docs # Install all pre-built wheels WHEELS=$(find wheels/ -name "*.whl") if [ -n "$WHEELS" ]; then echo "Installing wheels:" echo "$WHEELS" uv pip install wheels/*.whl else echo "ERROR: No wheels found!" exit 1 fi - name: Build docs run: | set -x cd docs # build.sh downloads the example data, registers the Jupyter kernel # myst-nb needs, symlinks the data next to each executed page, and # runs sphinx. Using it here keeps CI identical to a local build. uv run --no-project bash ./build.sh - name: Copy & push the generated HTML if: github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref_type == 'tag') run: | set -x cd docs-target # delete anything but: 1) '.'; 2) '..'; 3) .git/ find ./ | grep -vE "^./$|^../$|^./.git" | xargs rm -rf cp ../.asf.yaml . cp -r ../docs/build/html/* . git status --porcelain if [ "$(git status --porcelain)" != "" ]; then git config user.name "github-actions[bot]" git config user.email "github-actions[bot]@users.noreply.github.com" git add --all git commit -m 'Publish built docs triggered by ${{ github.sha }}' git push || git push --force fi apache-datafusion-python-54.0.0/.github/workflows/ci.yml000066400000000000000000000023731521727353300232230ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # CI workflow for pull requests - runs tests in DEBUG mode for faster feedback name: CI on: pull_request: branches: ["main"] concurrency: group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} cancel-in-progress: true jobs: build: uses: ./.github/workflows/build.yml with: build_mode: debug run_wheels: false secrets: inherit test: needs: build uses: ./.github/workflows/test.yml secrets: inherit apache-datafusion-python-54.0.0/.github/workflows/codeql.yml000066400000000000000000000030561521727353300240760ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # name: "CodeQL" on: push: branches: [ "main" ] pull_request: branches: [ "main" ] schedule: - cron: '16 4 * * 1' permissions: contents: read jobs: analyze: name: Analyze Actions runs-on: ubuntu-latest permissions: contents: read security-events: write steps: - name: Checkout repository uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6 with: persist-credentials: false - name: Initialize CodeQL uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4 with: languages: actions - name: Perform CodeQL Analysis uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4 with: category: "/language:actions" apache-datafusion-python-54.0.0/.github/workflows/dev.yml000066400000000000000000000021731521727353300234040ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. name: Dev on: [push, pull_request] jobs: rat: name: Release Audit Tool (RAT) runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v6 - name: Setup Python uses: actions/setup-python@v6 with: python-version: "3.14" - name: Audit licenses run: ./dev/release/run-rat.sh . apache-datafusion-python-54.0.0/.github/workflows/release.yml000066400000000000000000000027531521727353300242520ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # Release workflow - runs tests in RELEASE mode and builds distribution wheels # Triggered on: # - Merges to main # - Release candidate tags (*-rc*) # - Release tags (e.g., 45.0.0) name: Release Build on: push: branches: - "main" tags: - "*-rc*" # Release candidates (e.g., 45.0.0-rc1) - "[0-9]+.*" # Release tags (e.g., 45.0.0) concurrency: group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} cancel-in-progress: true jobs: build: uses: ./.github/workflows/build.yml with: build_mode: release run_wheels: true secrets: inherit test: needs: build uses: ./.github/workflows/test.yml secrets: inherit apache-datafusion-python-54.0.0/.github/workflows/take.yml000066400000000000000000000036001521727353300235460ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. name: Assign the issue via a `take` comment on: issue_comment: types: created permissions: issues: write jobs: issue_assign: runs-on: ubuntu-latest if: (!github.event.issue.pull_request) && github.event.comment.body == 'take' concurrency: group: ${{ github.actor }}-issue-assign steps: - run: | CODE=$(curl -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" -LI https://api.github.com/repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/assignees/${{ github.event.comment.user.login }} -o /dev/null -w '%{http_code}\n' -s) if [ "$CODE" -eq "204" ] then echo "Assigning issue ${{ github.event.issue.number }} to ${{ github.event.comment.user.login }}" curl -H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" -d '{"assignees": ["${{ github.event.comment.user.login }}"]}' https://api.github.com/repos/${{ github.repository }}/issues/${{ github.event.issue.number }}/assignees else echo "Cannot assign issue ${{ github.event.issue.number }} to ${{ github.event.comment.user.login }}" fiapache-datafusion-python-54.0.0/.github/workflows/test.yml000066400000000000000000000121361521727353300236050ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # Reusable workflow for running tests. # Single matrix covers both GIL (abi3 wheel) and free-threaded # (per-interpreter wheels) builds. name: Test on: workflow_call: env: UV_LOCKED: true jobs: test-matrix: runs-on: ubuntu-latest # Backstop: a hung multiprocessing worker (e.g. during a pickle regression) # should not block CI longer than this. timeout-minutes: 30 strategy: fail-fast: false matrix: include: # GIL builds — all share the same abi3 wheel. - { python-version: "3.10", wheel-tag: "abi3", freethreaded: false } - { python-version: "3.11", wheel-tag: "abi3", freethreaded: false } - { python-version: "3.12", wheel-tag: "abi3", freethreaded: false } - { python-version: "3.13", wheel-tag: "abi3", freethreaded: false } - { python-version: "3.14", wheel-tag: "abi3", freethreaded: false } # Free-threaded builds — one wheel per interpreter. - { python-version: "3.14t", wheel-tag: "3.14t", freethreaded: true } steps: - uses: actions/checkout@v6 - name: Setup Python id: setup-python uses: actions/setup-python@v6 with: python-version: ${{ matrix.python-version }} freethreaded: ${{ matrix.freethreaded }} - name: Cache Cargo uses: actions/cache@v5 with: path: ~/.cargo key: cargo-cache-stable-${{ hashFiles('Cargo.lock') }} - name: Install dependencies uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 with: enable-cache: true - name: Download pre-built Linux wheel uses: actions/download-artifact@v8 with: name: dist-manylinux-x86_64-${{ matrix.wheel-tag }} path: wheels/ # FFI test wheel only built once (under the abi3 matrix entry in build.yml). - name: Download pre-built FFI test wheel if: matrix.wheel-tag == 'abi3' uses: actions/download-artifact@v8 with: name: test-ffi-manylinux-x86_64 path: wheels/ - name: Install from pre-built wheels run: | set -x # Create the venv with the setup-python interpreter, then point # every uv command explicitly at the venv's own interpreter. # uv's interpreter discovery skips free-threaded builds unless # asked by exact path, so plain `uv sync` (even with an activated # venv or --active) re-picks the system 3.12 and recreates .venv. # Targeting .venv/bin/python keeps sync and pip install in the # same 3.14t environment as the cp314t wheel. uv venv --python "${{ steps.setup-python.outputs.python-path }}" VENV_PY="$PWD/.venv/bin/python" uv sync --python "$VENV_PY" --dev --no-install-package datafusion WHEELS=$(find wheels/ -name "*.whl") if [ -n "$WHEELS" ]; then echo "Installing wheels:" echo "$WHEELS" uv pip install --python "$VENV_PY" wheels/*.whl else echo "ERROR: No wheels found!" exit 1 fi - name: Run tests env: RUST_BACKTRACE: 1 # On free-threaded interpreters, fail loud if any C extension # re-enables the GIL implicitly. PYTHON_GIL: ${{ matrix.freethreaded && '0' || '' }} run: | git submodule update --init # Use the .venv interpreter directly; uv discovery would skip the # free-threaded build and re-pick the system 3.12 (see install step). uv run --python "$PWD/.venv/bin/python" --no-project pytest -v --import-mode=importlib # FFI + TPC-H examples only need to run once; gate to abi3 entries. - name: FFI unit tests if: matrix.wheel-tag == 'abi3' run: | cd examples/datafusion-ffi-example uv run --no-project pytest python/tests/_test*.py - name: Run tpchgen-cli to create 1 Gb dataset if: matrix.wheel-tag == 'abi3' run: | mkdir examples/tpch/data cd examples/tpch/data uv pip install tpchgen-cli uv run --no-project tpchgen-cli -s 1 --format=parquet - name: Run TPC-H examples if: matrix.wheel-tag == 'abi3' run: | cd examples/tpch uv run --no-project pytest _tests.py apache-datafusion-python-54.0.0/.github/workflows/verify-release-candidate.yml000066400000000000000000000047161521727353300274670ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. name: Verify Release Candidate # NOTE: This workflow is intended to be run manually via workflow_dispatch. on: workflow_dispatch: inputs: version: description: Version number (e.g., 52.0.0) required: true type: string rc_number: description: Release candidate number (e.g., 1) required: true type: string concurrency: group: ${{ github.repository }}-${{ github.ref }}-${{ github.workflow }} cancel-in-progress: true jobs: verify: name: Verify RC (${{ matrix.os }}-${{ matrix.arch }}) strategy: fail-fast: false matrix: include: # Linux - os: linux arch: x64 runner: ubuntu-latest - os: linux arch: arm64 runner: ubuntu-24.04-arm # macOS - os: macos arch: arm64 runner: macos-latest - os: macos arch: x64 runner: macos-15-intel # Windows - os: windows arch: x64 runner: windows-latest runs-on: ${{ matrix.runner }} steps: - name: Checkout repository uses: actions/checkout@v6 - name: Set up protoc uses: arduino/setup-protoc@v3 with: version: "27.4" repo-token: ${{ secrets.GITHUB_TOKEN }} - name: Set RUSTFLAGS for Windows GNU linker if: matrix.os == 'windows' shell: bash run: echo "RUSTFLAGS=-C link-arg=-Wl,--exclude-libs=ALL" >> "$GITHUB_ENV" - name: Run release candidate verification shell: bash run: ./dev/release/verify-release-candidate.sh "${{ inputs.version }}" "${{ inputs.rc_number }}" apache-datafusion-python-54.0.0/.gitignore000066400000000000000000000007441521727353300205000ustar00rootroot00000000000000target /venv .idea /docs/temp /docs/build .DS_Store .vscode # Byte-compiled / optimized / DLL files __pycache__/ *.py[cod] *$py.class # Python dist ignore dist # C extensions *.so # Python dist dist # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: .python-version venv .venv apache-rat-*.jar *rat.txt .env CHANGELOG.md.bak docs/mdbook/book .pyo3_build_config apache-datafusion-python-54.0.0/.gitmodules000066400000000000000000000002641521727353300206620ustar00rootroot00000000000000[submodule "testing"] path = testing url = https://github.com/apache/arrow-testing.git [submodule "parquet"] path = parquet url = https://github.com/apache/parquet-testing.git apache-datafusion-python-54.0.0/.pre-commit-config.yaml000066400000000000000000000044141521727353300227670ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. repos: - repo: https://github.com/rhysd/actionlint rev: v1.7.12 hooks: - id: actionlint-docker - repo: https://github.com/astral-sh/ruff-pre-commit # Ruff version. rev: v0.15.1 hooks: # Run the linter. - id: ruff # Run the formatter. - id: ruff-format - repo: local hooks: - id: rust-fmt name: Rust fmt description: Run cargo fmt on files included in the commit. rustfmt should be installed before-hand. entry: cargo +nightly fmt --all -- pass_filenames: true types: [file, rust] language: system - id: rust-clippy name: Rust clippy description: Run cargo clippy on files included in the commit. clippy should be installed before-hand. entry: cargo clippy --all-targets --all-features -- -Dclippy::all -D warnings -Aclippy::redundant_closure pass_filenames: false types: [file, rust] language: system - repo: https://github.com/codespell-project/codespell rev: v2.4.1 hooks: - id: codespell args: [ --toml, "pyproject.toml"] additional_dependencies: - tomli - repo: https://github.com/astral-sh/uv-pre-commit # uv version. rev: 0.10.7 hooks: # Update the uv lockfile - id: uv-lock default_language_version: python: python3 apache-datafusion-python-54.0.0/AGENTS.md000066400000000000000000000102201521727353300200010ustar00rootroot00000000000000 # Agent Instructions for Contributors This file is for agents working **on** the datafusion-python project (developing, testing, reviewing). If you need to **use** the DataFusion DataFrame API (write queries, build expressions, understand available functions), see the user-facing skill at [`SKILL.md`](skills/datafusion_python/SKILL.md). ## Skills This project uses AI agent skills stored in `.ai/skills/`. Each skill is a directory containing a `SKILL.md` file with instructions for performing a specific task. Skills follow the [Agent Skills](https://agentskills.io) open standard. Each skill directory contains: - `SKILL.md` — The skill definition with YAML frontmatter (name, description, argument-hint) and detailed instructions. - Additional supporting files as needed. To discover what skills are available, list `.ai/skills/` and read each `SKILL.md`. The frontmatter `name` and `description` fields summarize the skill's purpose. ## Pull Requests Every pull request must follow the template in `.github/pull_request_template.md`. The description must include these sections: 1. **Which issue does this PR close?** — Link the issue with `Closes #NNN`. 2. **Rationale for this change** — Why the change is needed (skip if the issue already explains it clearly). 3. **What changes are included in this PR?** — Summarize the individual changes. 4. **Are there any user-facing changes?** — Note any changes visible to users (new APIs, changed behavior, new files shipped in the package, etc.). If there are breaking changes to public APIs, add the `api change` label. ## Pre-commit Checks Always run pre-commit checks **before** committing. The hooks are defined in `.pre-commit-config.yaml` and run automatically on `git commit` if pre-commit is installed as a git hook. To run all hooks manually: ```bash pre-commit run --all-files ``` Fix any failures before committing. ## Python Function Docstrings Every Python function must include a docstring with usage examples. - **Examples are required**: Each function needs at least one doctest-style example demonstrating basic usage. - **Optional parameters**: If a function has optional parameters, include separate examples that show usage both without and with the optional arguments. Pass optional arguments using their keyword name (e.g., `step=dfn.lit(3)`) so readers can immediately see which parameter is being demonstrated. - **Reuse input data**: Use the same input data across examples wherever possible. The examples should demonstrate how different optional arguments change the output for the same input, making the effect of each option easy to understand. - **Alias functions**: Functions that are simple aliases (e.g., `list_sort` aliasing `array_sort`) only need a one-line description and a `See Also` reference to the primary function. They do not need their own examples. ## Aggregate and Window Function Documentation When adding or updating an aggregate or window function, ensure the corresponding site documentation is kept in sync: - **Aggregations**: `docs/source/user-guide/common-operations/aggregations.md` — add new aggregate functions to the "Aggregate Functions" list and include usage examples if appropriate. - **Window functions**: `docs/source/user-guide/common-operations/windows.md` — add new window functions to the "Available Functions" list and include usage examples if appropriate. apache-datafusion-python-54.0.0/CHANGELOG.md000066400000000000000000000015601521727353300203160ustar00rootroot00000000000000 # DataFusion Python Changelog The changelogs have now moved [here](./dev/changelog). apache-datafusion-python-54.0.0/CLAUDE.md000077700000000000000000000000001521727353300212602AGENTS.mdustar00rootroot00000000000000apache-datafusion-python-54.0.0/Cargo.lock000066400000000000000000003345241521727353300204230ustar00rootroot00000000000000# This file is automatically @generated by Cargo. # It is not intended for manual editing. version = 4 [[package]] name = "adler2" version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "320119579fcad9c21884f5c4861d16174d0e06250625266f50fe6898340abefa" [[package]] name = "ahash" version = "0.8.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a15f179cd60c4584b8a8c596927aadc462e27f2ca70c04e0071964a73ba7a75" dependencies = [ "cfg-if", "const-random", "getrandom 0.3.4", "once_cell", "version_check", "zerocopy", ] [[package]] name = "aho-corasick" version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ddd31a130427c27518df266943a5308ed92d4b226cc639f5a8f1002816174301" dependencies = [ "memchr", ] [[package]] name = "alloc-no-stdlib" version = "2.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" [[package]] name = "alloc-stdlib" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" dependencies = [ "alloc-no-stdlib", ] [[package]] name = "allocator-api2" version = "0.2.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923" [[package]] name = "android_system_properties" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "819e7219dbd41043ac279b19830f2efc897156490d7fd6ea916720117ee66311" dependencies = [ "libc", ] [[package]] name = "anyhow" version = "1.0.102" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" [[package]] name = "ar_archive_writer" version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4087686b4b0a3427190bae57a1d9a478dbb2d40c5dc1bd6e2b6d797913bdd348" dependencies = [ "object", ] [[package]] name = "arc-swap" version = "1.9.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a3a1fd6f75306b68087b831f025c712524bcb19aad54e557b1129cfa0a2b207" dependencies = [ "rustversion", ] [[package]] name = "arrayref" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" [[package]] name = "arrayvec" version = "0.7.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "arrow" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "378530e55cd479eda3c14eb345310799717e6f76d0c332041e8487022166b471" dependencies = [ "arrow-arith", "arrow-array", "arrow-buffer", "arrow-cast", "arrow-csv", "arrow-data", "arrow-ipc", "arrow-json", "arrow-ord", "arrow-pyarrow", "arrow-row", "arrow-schema", "arrow-select", "arrow-string", ] [[package]] name = "arrow-arith" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a0ab212d2c1886e802f51c5212d78ebbcbb0bec980fff9dadc1eb8d45cd0b738" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "chrono", "num-traits", ] [[package]] name = "arrow-array" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfd33d3e92f207444098c75b42de99d329562be0cf686b307b097cc52b4e999e" dependencies = [ "ahash", "arrow-buffer", "arrow-data", "arrow-schema", "chrono", "chrono-tz", "half", "hashbrown 0.17.1", "num-complex", "num-integer", "num-traits", ] [[package]] name = "arrow-avro" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "049230728cd6e093088c8d231b4beede184e35cad7777c1505c0d5a8571f4376" dependencies = [ "arrow-array", "arrow-buffer", "arrow-schema", "bytes", "bzip2", "crc", "flate2", "indexmap", "liblzma", "rand 0.9.4", "serde", "serde_json", "snap", "strum_macros", "uuid", "zstd", ] [[package]] name = "arrow-buffer" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0c6cd424c2693bcdbc150d843dc9d4d137dd2de4782ce6df491ad11a3a0416c0" dependencies = [ "bytes", "half", "num-bigint", "num-traits", ] [[package]] name = "arrow-cast" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4c5aefb56a2c02e9e2b30746241058b85f8983f0fcff2ba0c6d09006e1cded7f" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-ord", "arrow-schema", "arrow-select", "atoi", "base64", "chrono", "comfy-table", "half", "lexical-core", "num-traits", "ryu", ] [[package]] name = "arrow-csv" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e94e8cf7e517657a52b91ea1263acf38c4ca62a84655d72458a3359b12ab97de" dependencies = [ "arrow-array", "arrow-cast", "arrow-schema", "chrono", "csv", "csv-core", "regex", ] [[package]] name = "arrow-data" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3c88210023a2bfee1896af366309a3028fc3bcbd6515fa29a7990ee1baa08ee0" dependencies = [ "arrow-buffer", "arrow-schema", "half", "num-integer", "num-traits", ] [[package]] name = "arrow-ipc" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "238438f0834483703d88896db6fe5a7138b2230debc31b34c0336c2996e3c64f" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "arrow-select", "flatbuffers", "lz4_flex", "zstd", ] [[package]] name = "arrow-json" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "205ca2119e6d679d5c133c6f30e68f027738d95ed948cf77677ea69c7800036b" dependencies = [ "arrow-array", "arrow-buffer", "arrow-cast", "arrow-ord", "arrow-schema", "arrow-select", "chrono", "half", "indexmap", "itoa", "lexical-core", "memchr", "num-traits", "ryu", "serde_core", "serde_json", "simdutf8", ] [[package]] name = "arrow-ord" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1bffd8fd2579286a5d63bac898159873e5094a79009940bcb42bbfce4f19f1d0" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "arrow-select", ] [[package]] name = "arrow-pyarrow" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d29abdf672a81c1aeb57fd2661457f9918964d49aed0e9f18932535f2a9e49ce" dependencies = [ "arrow-array", "arrow-data", "arrow-schema", "pyo3", ] [[package]] name = "arrow-row" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bab5994731204603c73ba69267616c50f80780774c6bb0476f1f830625115e0c" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "half", ] [[package]] name = "arrow-schema" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f633dbfdf39c039ada1bf9e34c694816eb71fbb7dc78f613993b7245e078a1ed" dependencies = [ "bitflags", "serde_core", "serde_json", ] [[package]] name = "arrow-select" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8cd065c54172ac787cf3f2f8d4107e0d3fdc26edba76fdf4f4cc170258942222" dependencies = [ "ahash", "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "num-traits", ] [[package]] name = "arrow-string" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "29dd7cda3ab9692f43a2e4acc444d760cc17b12bb6d8232ddf64e9bab7c06b42" dependencies = [ "arrow-array", "arrow-buffer", "arrow-data", "arrow-schema", "arrow-select", "memchr", "num-traits", "regex", "regex-syntax", ] [[package]] name = "async-compression" version = "0.4.42" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e79b3f8a79cccc2898f31920fc69f304859b3bd567490f75ebf51ae1c792a9ac" dependencies = [ "compression-codecs", "compression-core", "pin-project-lite", "tokio", ] [[package]] name = "async-ffi" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f4de21c0feef7e5a556e51af767c953f0501f7f300ba785cc99c47bdc8081a50" [[package]] name = "async-recursion" version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "async-trait" version = "0.1.89" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "atoi" version = "2.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" dependencies = [ "num-traits", ] [[package]] name = "atomic-waker" version = "1.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1505bd5d3d116872e7271a6d4e16d81d0c8570876c8de68093a09ac269d8aac0" [[package]] name = "autocfg" version = "1.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53" [[package]] name = "base64" version = "0.22.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6" [[package]] name = "bigdecimal" version = "0.4.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4d6867f1565b3aad85681f1015055b087fcfd840d6aeee6eee7f2da317603695" dependencies = [ "autocfg", "libm", "num-bigint", "num-integer", "num-traits", ] [[package]] name = "bitflags" version = "2.13.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4388bee8683e3d04af747c73422af53102d2bd24d9eadb6cbc100baef4b43f8" [[package]] name = "blake2" version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "46502ad458c9a52b69d4d4d32775c788b7a1b85e8bc9d482d92250fc0e3f8efe" dependencies = [ "digest 0.10.7", ] [[package]] name = "blake3" version = "1.8.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0aa83c34e62843d924f905e0f5c866eb1dd6545fc4d719e803d9ba6030371fce" dependencies = [ "arrayref", "arrayvec", "cc", "cfg-if", "constant_time_eq", "cpufeatures", ] [[package]] name = "block-buffer" version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71" dependencies = [ "generic-array", ] [[package]] name = "block-buffer" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa" dependencies = [ "hybrid-array", ] [[package]] name = "brotli" version = "8.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5cc91aac060a7a1e25823bdccbfb6af1875b88f17c6daac97894eed8207166b3" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", "brotli-decompressor", ] [[package]] name = "brotli-decompressor" version = "5.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", ] [[package]] name = "bumpalo" version = "3.20.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649" [[package]] name = "byteorder" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" [[package]] name = "bzip2" version = "0.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f3a53fac24f34a81bc9954b5d6cfce0c21e18ec6959f44f56e8e90e4bb7c346c" dependencies = [ "libbz2-rs-sys", ] [[package]] name = "cc" version = "1.2.64" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dad887fd958be91b5098c0248def011f4523ab786cd411be668777e55063501f" dependencies = [ "find-msvc-tools", "jobserver", "libc", "shlex", ] [[package]] name = "cfg-if" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801" [[package]] name = "cfg_aliases" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chacha20" version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" dependencies = [ "cfg-if", "cpufeatures", "rand_core 0.10.1", ] [[package]] name = "chrono" version = "0.4.45" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1aa79e62e7697b8e29b513a68abacf485adcd1fe8284a4316c5ae868e6633327" dependencies = [ "iana-time-zone", "num-traits", "serde", "windows-link", ] [[package]] name = "chrono-tz" version = "0.10.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6139a8597ed92cf816dfb33f5dd6cf0bb93a6adc938f11039f371bc5bcd26c3" dependencies = [ "chrono", "phf", ] [[package]] name = "cmake" version = "0.1.58" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" dependencies = [ "cc", ] [[package]] name = "comfy-table" version = "7.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "958c5d6ecf1f214b4c2bbbbf6ab9523a864bd136dcf71a7e8904799acfe1ad47" dependencies = [ "unicode-segmentation", "unicode-width", ] [[package]] name = "compression-codecs" version = "0.4.38" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce2548391e9c1929c21bf6aa2680af86fe4c1b33e6cea9ac1cfeec0bd11218cf" dependencies = [ "bzip2", "compression-core", "flate2", "liblzma", "memchr", "zstd", "zstd-safe", ] [[package]] name = "compression-core" version = "0.4.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cc14f565cf027a105f7a44ccf9e5b424348421a1d8952a8fc9d499d313107789" [[package]] name = "const-oid" version = "0.10.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" [[package]] name = "const-random" version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "87e00182fe74b066627d63b85fd550ac2998d4b0bd86bfed477a0ae4c7c71359" dependencies = [ "const-random-macro", ] [[package]] name = "const-random-macro" version = "0.1.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f9d839f2a20b0aee515dc581a6172f2321f96cab76c1a38a4c584a194955390e" dependencies = [ "getrandom 0.2.17", "once_cell", "tiny-keccak", ] [[package]] name = "constant_time_eq" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" [[package]] name = "core-foundation" version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b2a6cd9ae233e7f62ba4e9353e81a88df7fc8a5987b8d445b4d90c879bd156f6" dependencies = [ "core-foundation-sys", "libc", ] [[package]] name = "core-foundation-sys" version = "0.8.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773648b94d0e5d620f64f280777445740e61fe701025087ec8b57f45c791888b" [[package]] name = "cpufeatures" version = "0.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" dependencies = [ "libc", ] [[package]] name = "crc" version = "3.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5eb8a2a1cd12ab0d987a5d5e825195d372001a4094a0376319d5a0ad71c1ba0d" dependencies = [ "crc-catalog", ] [[package]] name = "crc-catalog" version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "217698eaf96b4a3f0bc4f3662aaa55bdf913cd54d7204591faa790070c6d0853" [[package]] name = "crc32fast" version = "1.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9481c1c90cbf2ac953f07c8d4a58aa3945c425b7185c9154d67a65e4230da511" dependencies = [ "cfg-if", ] [[package]] name = "crossbeam-utils" version = "0.8.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0a5c400df2834b80a4c3327b3aad3a4c4cd4de0629063962b03235697506a28" [[package]] name = "crunchy" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5" [[package]] name = "crypto-common" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a" dependencies = [ "generic-array", "typenum", ] [[package]] name = "crypto-common" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453" dependencies = [ "hybrid-array", ] [[package]] name = "cstr" version = "0.2.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68523903c8ae5aacfa32a0d9ae60cadeb764e1da14ee0d26b1f3089f13a54636" dependencies = [ "proc-macro2", "quote", ] [[package]] name = "csv" version = "1.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52cd9d68cf7efc6ddfaaee42e7288d3a99d613d4b50f76ce9827ae0c6e14f938" dependencies = [ "csv-core", "itoa", "ryu", "serde_core", ] [[package]] name = "csv-core" version = "0.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "704a3c26996a80471189265814dbc2c257598b96b8a7feae2d31ace646bb9782" dependencies = [ "memchr", ] [[package]] name = "dashmap" version = "6.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6361d5c062261c78a176addb82d4c821ae42bed6089de0e12603cd25de2059c" dependencies = [ "cfg-if", "crossbeam-utils", "hashbrown 0.14.5", "lock_api", "once_cell", "parking_lot_core", ] [[package]] name = "datafusion" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "997a31e15872606a49478e670c58302094c97cb96abb0a7d60720f8e92170040" dependencies = [ "arrow", "arrow-schema", "async-trait", "bzip2", "chrono", "datafusion-catalog", "datafusion-catalog-listing", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-datasource-arrow", "datafusion-datasource-avro", "datafusion-datasource-csv", "datafusion-datasource-json", "datafusion-datasource-parquet", "datafusion-execution", "datafusion-expr", "datafusion-expr-common", "datafusion-functions", "datafusion-functions-aggregate", "datafusion-functions-nested", "datafusion-functions-table", "datafusion-functions-window", "datafusion-optimizer", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-session", "datafusion-sql", "flate2", "futures", "indexmap", "itertools", "liblzma", "log", "object_store", "parking_lot", "parquet", "sqlparser", "tempfile", "tokio", "url", "uuid", "zstd", ] [[package]] name = "datafusion-catalog" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f7dd61161508f8f5fa1107774ea687bd753c22d83a32eebf963549f89de14139" dependencies = [ "arrow", "async-trait", "dashmap", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", "datafusion-physical-plan", "datafusion-session", "futures", "itertools", "log", "object_store", "parking_lot", "tokio", ] [[package]] name = "datafusion-catalog-listing" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "897c70f871277f9ce99aa38347be0d679bbe3e617156c4d2a8378cec8a2a0891" dependencies = [ "arrow", "async-trait", "datafusion-catalog", "datafusion-common", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", "futures", "itertools", "log", "object_store", ] [[package]] name = "datafusion-common" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "121c9ded5d87d9172319e006f2afdb9928d72dbacd6a90a458d8acb1e3b43a65" dependencies = [ "arrow", "arrow-ipc", "arrow-schema", "chrono", "foldhash 0.2.0", "half", "hashbrown 0.17.1", "indexmap", "itertools", "libc", "log", "object_store", "parquet", "recursive", "sqlparser", "tokio", "uuid", "web-time", ] [[package]] name = "datafusion-common-runtime" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "981b9dae74f78ee3d9f714fb49b01919eab975461b56149510c3ba9ea11287d1" dependencies = [ "futures", "log", "tokio", ] [[package]] name = "datafusion-datasource" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ffd7d295b2ec7c00d8a56562f41ed41062cf0af75549ed891c12a0a09eddfefe" dependencies = [ "arrow", "async-compression", "async-trait", "bytes", "bzip2", "chrono", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "flate2", "futures", "glob", "itertools", "liblzma", "log", "object_store", "parking_lot", "rand 0.9.4", "tokio", "tokio-util", "url", "zstd", ] [[package]] name = "datafusion-datasource-arrow" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "552b0b3f342f7ec41b3fbd70f6339dc82a30cfd0349e7f280e7852528085349f" dependencies = [ "arrow", "arrow-ipc", "async-trait", "bytes", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "futures", "itertools", "object_store", "tokio", ] [[package]] name = "datafusion-datasource-avro" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fb517d08967d536284ce70afb5fe8583133779249f2d7b90587d339741a7f195" dependencies = [ "arrow", "arrow-avro", "async-trait", "bytes", "datafusion-common", "datafusion-datasource", "datafusion-physical-expr-adapter", "datafusion-physical-plan", "datafusion-session", "futures", "object_store", ] [[package]] name = "datafusion-datasource-csv" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68850aa426b897e879c8b87e512ea8124f1d0a2869a4e51808ddaaddf1bc0ada" dependencies = [ "arrow", "async-trait", "bytes", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "futures", "object_store", "regex", "tokio", ] [[package]] name = "datafusion-datasource-json" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "402f93242ae08ef99139ee2c528a49d087efe88d5c7b2c3ff5480855a40ce54f" dependencies = [ "arrow", "async-trait", "bytes", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-session", "futures", "object_store", "tokio", "tokio-stream", ] [[package]] name = "datafusion-datasource-parquet" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ffd2499c1bee0eeccf6a57156105700eeeb17bc701899ac719183c4e74231450" dependencies = [ "arrow", "async-trait", "bytes", "datafusion-common", "datafusion-common-runtime", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-functions", "datafusion-functions-aggregate-common", "datafusion-physical-expr", "datafusion-physical-expr-adapter", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-pruning", "datafusion-session", "futures", "itertools", "log", "object_store", "parking_lot", "parquet", "tokio", ] [[package]] name = "datafusion-doc" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cb9e7e5d11130c48c8bd4e80c79a9772dd28ce6dc330baca9246205d245b9e2e" [[package]] name = "datafusion-execution" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "37a8643ab852eb68864e1b72ae789e8066282dce48eea6347ffb0aee33d1ccc0" dependencies = [ "arrow", "arrow-buffer", "async-trait", "dashmap", "datafusion-common", "datafusion-expr", "datafusion-physical-expr-common", "futures", "log", "object_store", "parking_lot", "rand 0.9.4", "tempfile", "url", ] [[package]] name = "datafusion-expr" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6932f4d71eed9c8d9341476a2b845aadfabde5495d08dbcd8fc23881f49fa7a0" dependencies = [ "arrow", "arrow-schema", "async-trait", "chrono", "datafusion-common", "datafusion-doc", "datafusion-expr-common", "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", "indexmap", "itertools", "recursive", "serde_json", "sqlparser", ] [[package]] name = "datafusion-expr-common" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0225491839a31b1f7d2cb8092c2d50792e2fe1c1724e4e6d08e011f5feaf4ed2" dependencies = [ "arrow", "datafusion-common", "indexmap", "itertools", ] [[package]] name = "datafusion-ffi" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5660e8fa79fd51e29ce46f3026b67317ef738ebd633e106beb1a1907a406152" dependencies = [ "arrow", "arrow-schema", "async-ffi", "async-trait", "chrono", "datafusion-catalog", "datafusion-common", "datafusion-datasource", "datafusion-execution", "datafusion-expr", "datafusion-functions-aggregate-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-optimizer", "datafusion-physical-plan", "datafusion-proto", "datafusion-proto-common", "datafusion-session", "futures", "libloading", "log", "prost", "semver", "stabby", "tokio", ] [[package]] name = "datafusion-ffi-example" version = "54.0.0" dependencies = [ "arrow", "arrow-array", "arrow-schema", "async-trait", "datafusion", "datafusion-catalog", "datafusion-common", "datafusion-expr", "datafusion-ffi", "datafusion-functions-aggregate", "datafusion-functions-window", "datafusion-proto", "datafusion-python-util", "pyo3", "pyo3-build-config", "pyo3-log", ] [[package]] name = "datafusion-functions" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "14872c47bfc3d21e53ec82f57074e6987a15941c1e2f43cde4ac6ae2746634e3" dependencies = [ "arrow", "arrow-buffer", "base64", "blake2", "blake3", "chrono", "chrono-tz", "datafusion-common", "datafusion-doc", "datafusion-execution", "datafusion-expr", "datafusion-expr-common", "datafusion-macros", "datafusion-physical-expr-common", "hex", "itertools", "log", "md-5 0.11.0", "memchr", "num-traits", "rand 0.9.4", "regex", "sha2", "uuid", ] [[package]] name = "datafusion-functions-aggregate" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75a2ca14e1b609be21e657e2d3130b2f446456b08393b377bb721a33952d2e09" dependencies = [ "arrow", "datafusion-common", "datafusion-doc", "datafusion-execution", "datafusion-expr", "datafusion-functions-aggregate-common", "datafusion-macros", "datafusion-physical-expr", "datafusion-physical-expr-common", "foldhash 0.2.0", "half", "log", "num-traits", ] [[package]] name = "datafusion-functions-aggregate-common" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ece74ba09092d2ef9c9b54a38445450aea292a1f8b04faf531936b723a24b3c" dependencies = [ "arrow", "datafusion-common", "datafusion-expr-common", "datafusion-physical-expr-common", ] [[package]] name = "datafusion-functions-nested" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f3e3f9ee8ca59bf70518802107de6f1b88a9509efdc629fadc5de9d6b2d5ef5" dependencies = [ "arrow", "arrow-ord", "datafusion-common", "datafusion-doc", "datafusion-execution", "datafusion-expr", "datafusion-expr-common", "datafusion-functions", "datafusion-functions-aggregate", "datafusion-functions-aggregate-common", "datafusion-macros", "datafusion-physical-expr-common", "hashbrown 0.17.1", "itertools", "itoa", "log", "memchr", ] [[package]] name = "datafusion-functions-table" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "89161dffc22cf2b50f9f4b1bee83b5221d3b4ed7c2e37fd7aa2b22a5297b3a26" dependencies = [ "arrow", "async-trait", "datafusion-catalog", "datafusion-common", "datafusion-expr", "datafusion-physical-expr", "datafusion-physical-plan", "parking_lot", ] [[package]] name = "datafusion-functions-window" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7339345b226b3874037708bf5023ba1c2de705128f8457a095aae5ae9cb9c78" dependencies = [ "arrow", "datafusion-common", "datafusion-doc", "datafusion-expr", "datafusion-functions-window-common", "datafusion-macros", "datafusion-physical-expr", "datafusion-physical-expr-common", "log", ] [[package]] name = "datafusion-functions-window-common" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fa84836dc2392df6f43d6a29d37fb56a8ebdc8b3f4e10ae8dc15861fd20278fb" dependencies = [ "datafusion-common", "datafusion-physical-expr-common", ] [[package]] name = "datafusion-macros" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "587164e03ad68732aa9e7bfe5686e3f25970d4c64fd4bd80790749840892dae5" dependencies = [ "datafusion-doc", "quote", "syn", ] [[package]] name = "datafusion-optimizer" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77f20e8cf9e8654d92f4c16b24c487353ee5bf153ffc12d5772cd399ab8cd281" dependencies = [ "arrow", "chrono", "datafusion-common", "datafusion-expr", "datafusion-expr-common", "datafusion-physical-expr", "indexmap", "itertools", "log", "recursive", "regex", "regex-syntax", ] [[package]] name = "datafusion-physical-expr" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f015a4a82f6f7ff7e1d8d4bf3870a936752fa38b17705dfcc14adef95aa8922c" dependencies = [ "arrow", "datafusion-common", "datafusion-expr", "datafusion-expr-common", "datafusion-functions-aggregate-common", "datafusion-physical-expr-common", "half", "hashbrown 0.17.1", "indexmap", "itertools", "parking_lot", "petgraph", "recursive", "tokio", ] [[package]] name = "datafusion-physical-expr-adapter" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "51e6ffff8acdfe54e0ea15ccf38115c4a9184433b0439f42907637928d00a235" dependencies = [ "arrow", "datafusion-common", "datafusion-expr", "datafusion-functions", "datafusion-physical-expr", "datafusion-physical-expr-common", "itertools", ] [[package]] name = "datafusion-physical-expr-common" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7967a3e171c6a4bf09474b3f7a14f1a3db13ed1714ba12156f33fcce2bba54e8" dependencies = [ "arrow", "chrono", "datafusion-common", "datafusion-expr-common", "hashbrown 0.17.1", "indexmap", "itertools", "parking_lot", "pin-project", ] [[package]] name = "datafusion-physical-optimizer" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "59ff803e2a96054cb6d83f35f9e60fd4f42eac515e1932bd1b2dbc91d5fcbf36" dependencies = [ "arrow", "datafusion-common", "datafusion-execution", "datafusion-expr", "datafusion-expr-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-pruning", "itertools", "recursive", ] [[package]] name = "datafusion-physical-plan" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "776ee54d47d15bdb126452f9ca17b03761e3b004682914beaedd3f86eb507fbc" dependencies = [ "arrow", "arrow-data", "arrow-ipc", "arrow-ord", "arrow-schema", "async-trait", "datafusion-common", "datafusion-common-runtime", "datafusion-execution", "datafusion-expr", "datafusion-functions", "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "futures", "half", "hashbrown 0.17.1", "indexmap", "itertools", "log", "num-traits", "parking_lot", "pin-project-lite", "tokio", ] [[package]] name = "datafusion-proto" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9dd15a1ba5d3af93808241065c6c44dbca8296a189845e8a587c45c07bf0ffae" dependencies = [ "arrow", "chrono", "datafusion-catalog", "datafusion-catalog-listing", "datafusion-common", "datafusion-datasource", "datafusion-datasource-arrow", "datafusion-datasource-csv", "datafusion-datasource-json", "datafusion-datasource-parquet", "datafusion-execution", "datafusion-expr", "datafusion-functions-table", "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "datafusion-proto-common", "object_store", "prost", ] [[package]] name = "datafusion-proto-common" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "90042982cf9462eb06a0b81f92efa4188dae871e7ea3ab8dc61aa9c9349b2530" dependencies = [ "arrow", "datafusion-common", "prost", ] [[package]] name = "datafusion-pruning" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d5fb9e5774660aa69c3ba93c610f175f75b65cb8c3776edb3626de8f3a4f4ee3" dependencies = [ "arrow", "datafusion-common", "datafusion-datasource", "datafusion-expr-common", "datafusion-physical-expr", "datafusion-physical-expr-common", "datafusion-physical-plan", "log", ] [[package]] name = "datafusion-python" version = "54.0.0" dependencies = [ "arrow", "arrow-select", "async-trait", "chrono", "cstr", "datafusion", "datafusion-ffi", "datafusion-proto", "datafusion-python-util", "datafusion-spark", "datafusion-substrait", "futures", "log", "mimalloc", "object_store", "parking_lot", "prost", "prost-types", "pyo3", "pyo3-async-runtimes", "pyo3-build-config", "pyo3-log", "serde_json", "tokio", "url", "uuid", ] [[package]] name = "datafusion-python-util" version = "54.0.0" dependencies = [ "arrow", "datafusion", "datafusion-ffi", "datafusion-proto", "prost", "pyo3", "tokio", ] [[package]] name = "datafusion-session" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "15ce715fa2a61f4623cc234bcc14a3ef6a91f189128d5b14b468a6a17cdfc417" dependencies = [ "async-trait", "datafusion-common", "datafusion-execution", "datafusion-expr", "datafusion-physical-plan", "parking_lot", ] [[package]] name = "datafusion-spark" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "390bb0bf37cb2b95ffd65eacd66f60df50793d1f94097799e416f39477a51957" dependencies = [ "arrow", "bigdecimal", "chrono", "crc32fast", "datafusion-catalog", "datafusion-common", "datafusion-execution", "datafusion-expr", "datafusion-functions", "datafusion-functions-aggregate", "datafusion-functions-aggregate-common", "datafusion-functions-nested", "log", "num-traits", "percent-encoding", "rand 0.9.4", "serde_json", "sha1", "sha2", "twox-hash", "url", ] [[package]] name = "datafusion-sql" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6094ad36a3ed6d7ac87b20b479b2d0b118250f66cf997603829fdc65b44a7099" dependencies = [ "arrow", "bigdecimal", "chrono", "datafusion-common", "datafusion-expr", "datafusion-functions-nested", "indexmap", "log", "recursive", "regex", "sqlparser", ] [[package]] name = "datafusion-substrait" version = "54.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b22c8f8c72d317e54fad6f85c0ef6d1e1da53cc7faadc7eea8daf0f8d86d4f2" dependencies = [ "async-recursion", "async-trait", "chrono", "datafusion", "half", "itertools", "object_store", "pbjson-types", "prost", "substrait", "tokio", "url", ] [[package]] name = "digest" version = "0.10.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer 0.10.4", "crypto-common 0.1.7", "subtle", ] [[package]] name = "digest" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2" dependencies = [ "block-buffer 0.12.1", "const-oid", "crypto-common 0.2.2", ] [[package]] name = "displaydoc" version = "0.2.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "dyn-clone" version = "1.0.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555" [[package]] name = "either" version = "1.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91622ff5e7162018101f2fea40d6ebf4a78bbe5a49736a2020649edf9693679e" [[package]] name = "equivalent" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" [[package]] name = "errno" version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb" dependencies = [ "libc", "windows-sys 0.61.2", ] [[package]] name = "fastrand" version = "2.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f1f227452a390804cdb637b74a86990f2a7d7ba4b7d5693aac9b4dd6defd8d6" [[package]] name = "find-msvc-tools" version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582" [[package]] name = "fixedbitset" version = "0.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d674e81391d1e1ab681a28d99df07927c6d4aa5b027d7da16ba32d1d21ecd99" [[package]] name = "flatbuffers" version = "25.12.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "35f6839d7b3b98adde531effaf34f0c2badc6f4735d26fe74709d8e513a96ef3" dependencies = [ "bitflags", "rustc_version", ] [[package]] name = "flate2" version = "1.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "843fba2746e448b37e26a819579957415c8cef339bf08564fe8b7ddbd959573c" dependencies = [ "crc32fast", "miniz_oxide", "zlib-rs", ] [[package]] name = "fnv" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1" [[package]] name = "foldhash" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2" [[package]] name = "foldhash" version = "0.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb" [[package]] name = "form_urlencoded" version = "1.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cb4cb245038516f5f85277875cdaa4f7d2c9a0fa0468de06ed190163b1581fcf" dependencies = [ "percent-encoding", ] [[package]] name = "futures" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8b147ee9d1f6d097cef9ce628cd2ee62288d963e16fb287bd9286455b241382d" dependencies = [ "futures-channel", "futures-core", "futures-executor", "futures-io", "futures-sink", "futures-task", "futures-util", ] [[package]] name = "futures-channel" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "07bbe89c50d7a535e539b8c17bc0b49bdb77747034daa8087407d655f3f7cc1d" dependencies = [ "futures-core", "futures-sink", ] [[package]] name = "futures-core" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e3450815272ef58cec6d564423f6e755e25379b217b0bc688e295ba24df6b1d" [[package]] name = "futures-executor" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "baf29c38818342a3b26b5b923639e7b1f4a61fc5e76102d4b1981c6dc7a7579d" dependencies = [ "futures-core", "futures-task", "futures-util", ] [[package]] name = "futures-io" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cecba35d7ad927e23624b22ad55235f2239cfa44fd10428eecbeba6d6a717718" [[package]] name = "futures-macro" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "futures-sink" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c39754e157331b013978ec91992bde1ac089843443c49cbc7f46150b0fad0893" [[package]] name = "futures-task" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "037711b3d59c33004d3856fbdc83b99d4ff37a24768fa1be9ce3538a1cde4393" [[package]] name = "futures-util" version = "0.3.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "389ca41296e6190b48053de0321d02a77f32f8a5d2461dd38762c0593805c6d6" dependencies = [ "futures-channel", "futures-core", "futures-io", "futures-macro", "futures-sink", "futures-task", "memchr", "pin-project-lite", "slab", ] [[package]] name = "generic-array" version = "0.14.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a" dependencies = [ "typenum", "version_check", ] [[package]] name = "getrandom" version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff2abc00be7fca6ebc474524697ae276ad847ad0a6b3faa4bcb027e9a4614ad0" dependencies = [ "cfg-if", "js-sys", "libc", "wasi", "wasm-bindgen", ] [[package]] name = "getrandom" version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "899def5c37c4fd7b2664648c28120ecec138e4d395b459e5ca34f9cce2dd77fd" dependencies = [ "cfg-if", "js-sys", "libc", "r-efi 5.3.0", "wasip2", "wasm-bindgen", ] [[package]] name = "getrandom" version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", "libc", "r-efi 6.0.0", "rand_core 0.10.1", ] [[package]] name = "glob" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0cc23270f6e1808e30a928bdc84dea0b9b4136a8bc82338574f23baf47bbd280" [[package]] name = "h2" version = "0.4.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155" dependencies = [ "atomic-waker", "bytes", "fnv", "futures-core", "futures-sink", "http", "indexmap", "slab", "tokio", "tokio-util", "tracing", ] [[package]] name = "half" version = "2.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b" dependencies = [ "cfg-if", "crunchy", "num-traits", "zerocopy", ] [[package]] name = "hashbrown" version = "0.14.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e5274423e17b7c9fc20b6e7e208532f9b19825d82dfd615708b70edd83df41f1" [[package]] name = "hashbrown" version = "0.15.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1" dependencies = [ "foldhash 0.1.5", ] [[package]] name = "hashbrown" version = "0.16.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100" dependencies = [ "allocator-api2", "equivalent", "foldhash 0.2.0", ] [[package]] name = "hashbrown" version = "0.17.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a" dependencies = [ "allocator-api2", "equivalent", "foldhash 0.2.0", ] [[package]] name = "heck" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2304e00983f87ffb38b55b444b5e3b60a884b5d30c0fca7d82fe33449bbe55ea" [[package]] name = "hex" version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f24254aa9a54b5c858eaee2f5bccdb46aaf0e486a595ed5fd8f86ba55232a70" [[package]] name = "http" version = "1.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6970f50e31d6fc17d3fa27329444bfa74e196cf62e95052a3f6fee181dba6425" dependencies = [ "bytes", "itoa", ] [[package]] name = "http-body" version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1efedce1fb8e6913f23e0c92de8e62cd5b772a67e7b3946df930a62566c93184" dependencies = [ "bytes", "http", ] [[package]] name = "http-body-util" version = "0.1.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b021d93e26becf5dc7e1b75b1bed1fd93124b374ceb73f43d4d4eafec896a64a" dependencies = [ "bytes", "futures-core", "http", "http-body", "pin-project-lite", ] [[package]] name = "httparse" version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6dbf3de79e51f3d586ab4cb9d5c3e2c14aa28ed23d180cf89b4df0454a69cc87" [[package]] name = "humantime" version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" [[package]] name = "hybrid-array" version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9155a582abd142abc056962c29e3ce5ff2ad5469f4246b537ed42c5deba857da" dependencies = [ "typenum", ] [[package]] name = "hyper" version = "1.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "55281c53a1894c864990125767da440a4e630446785086f52523b20033b74498" dependencies = [ "atomic-waker", "bytes", "futures-channel", "futures-core", "h2", "http", "http-body", "httparse", "itoa", "pin-project-lite", "smallvec", "tokio", "want", ] [[package]] name = "hyper-rustls" version = "0.27.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "33ca68d021ef39cf6463ab54c1d0f5daf03377b70561305bb89a8f83aab66e0f" dependencies = [ "http", "hyper", "hyper-util", "rustls", "rustls-native-certs", "tokio", "tokio-rustls", "tower-service", ] [[package]] name = "hyper-util" version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "96547c2556ec9d12fb1578c4eaf448b04993e7fb79cbaad930a656880a6bdfa0" dependencies = [ "base64", "bytes", "futures-channel", "futures-util", "http", "http-body", "hyper", "ipnet", "libc", "percent-encoding", "pin-project-lite", "socket2", "tokio", "tower-service", "tracing", ] [[package]] name = "iana-time-zone" version = "0.1.65" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e31bc9ad994ba00e440a8aa5c9ef0ec67d5cb5e5cb0cc7f8b744a35b389cc470" dependencies = [ "android_system_properties", "core-foundation-sys", "iana-time-zone-haiku", "js-sys", "log", "wasm-bindgen", "windows-core", ] [[package]] name = "iana-time-zone-haiku" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f31827a206f56af32e590ba56d5d2d085f558508192593743f16b2306495269f" dependencies = [ "cc", ] [[package]] name = "icu_collections" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c" dependencies = [ "displaydoc", "potential_utf", "utf8_iter", "yoke", "zerofrom", "zerovec", ] [[package]] name = "icu_locale_core" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29" dependencies = [ "displaydoc", "litemap", "tinystr", "writeable", "zerovec", ] [[package]] name = "icu_normalizer" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4" dependencies = [ "icu_collections", "icu_normalizer_data", "icu_properties", "icu_provider", "smallvec", "zerovec", ] [[package]] name = "icu_normalizer_data" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38" [[package]] name = "icu_properties" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de" dependencies = [ "icu_collections", "icu_locale_core", "icu_properties_data", "icu_provider", "zerotrie", "zerovec", ] [[package]] name = "icu_properties_data" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14" [[package]] name = "icu_provider" version = "2.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421" dependencies = [ "displaydoc", "icu_locale_core", "writeable", "yoke", "zerofrom", "zerotrie", "zerovec", ] [[package]] name = "idna" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b0875f23caa03898994f6ddc501886a45c7d3d62d04d2d90788d47be1b1e4de" dependencies = [ "idna_adapter", "smallvec", "utf8_iter", ] [[package]] name = "idna_adapter" version = "1.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cb68373c0d6620ef8105e855e7745e18b0d00d3bdb07fb532e434244cdb9a714" dependencies = [ "icu_normalizer", "icu_properties", ] [[package]] name = "indexmap" version = "2.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9" dependencies = [ "equivalent", "hashbrown 0.17.1", "serde", "serde_core", ] [[package]] name = "integer-encoding" version = "3.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" [[package]] name = "ipnet" version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2" [[package]] name = "itertools" version = "0.14.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285" dependencies = [ "either", ] [[package]] name = "itoa" version = "1.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f42a60cbdf9a97f5d2305f08a87dc4e09308d1276d28c869c684d7777685682" [[package]] name = "jobserver" version = "0.1.34" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9afb3de4395d6b3e67a780b6de64b51c978ecf11cb9a462c66be7d4ca9039d33" dependencies = [ "getrandom 0.3.4", "libc", ] [[package]] name = "js-sys" version = "0.3.102" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "03d04c30968dffe80775bd4d7fb676131cd04a1fb46d2686dbffbaec2d9dfd31" dependencies = [ "cfg-if", "futures-util", "wasm-bindgen", ] [[package]] name = "lexical-core" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7d8d125a277f807e55a77304455eb7b1cb52f2b18c143b60e766c120bd64a594" dependencies = [ "lexical-parse-float", "lexical-parse-integer", "lexical-util", "lexical-write-float", "lexical-write-integer", ] [[package]] name = "lexical-parse-float" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52a9f232fbd6f550bc0137dcb5f99ab674071ac2d690ac69704593cb4abbea56" dependencies = [ "lexical-parse-integer", "lexical-util", ] [[package]] name = "lexical-parse-integer" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9a7a039f8fb9c19c996cd7b2fcce303c1b2874fe1aca544edc85c4a5f8489b34" dependencies = [ "lexical-util", ] [[package]] name = "lexical-util" version = "1.0.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2604dd126bb14f13fb5d1bd6a66155079cb9fa655b37f875b3a742c705dbed17" [[package]] name = "lexical-write-float" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "50c438c87c013188d415fbabbb1dceb44249ab81664efbd31b14ae55dabb6361" dependencies = [ "lexical-util", "lexical-write-integer", ] [[package]] name = "lexical-write-integer" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "409851a618475d2d5796377cad353802345cba92c867d9fbcde9cf4eac4e14df" dependencies = [ "lexical-util", ] [[package]] name = "libbz2-rs-sys" version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "34b357333733e8260735ba5894eb928c02ecc69c78715f01a8019e7fa7f2db4c" [[package]] name = "libc" version = "0.2.186" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68ab91017fe16c622486840e4c83c9a37afeff978bd239b5293d61ece587de66" [[package]] name = "libloading" version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "754ca22de805bb5744484a5b151a9e1a8e837d5dc232c2d7d8c2e3492edc8b60" dependencies = [ "cfg-if", "windows-link", ] [[package]] name = "liblzma" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6033b77c21d1f56deeae8014eb9fbe7bdf1765185a6c508b5ca82eeaed7f899" dependencies = [ "liblzma-sys", ] [[package]] name = "liblzma-sys" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1a60851d15cd8c5346eca4ab8babff585be2ae4bc8097c067291d3ffe2add3b6" dependencies = [ "cc", "libc", "pkg-config", ] [[package]] name = "libm" version = "0.2.16" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981" [[package]] name = "libmimalloc-sys" version = "0.1.49" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a45a52f43e1c16f667ccfe4dd8c85b7f7c204fd5e3bf46c5b0db9a5c3c0b8e9" dependencies = [ "cc", ] [[package]] name = "linux-raw-sys" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53" [[package]] name = "litemap" version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0" [[package]] name = "lock_api" version = "0.4.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "224399e74b87b5f3557511d98dff8b14089b3dadafcab6bb93eab67d3aace965" dependencies = [ "scopeguard", ] [[package]] name = "log" version = "0.4.32" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" [[package]] name = "lru-slab" version = "0.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "112b39cec0b298b6c1999fee3e31427f74f676e4cb9879ed1a121b43661a4154" [[package]] name = "lz4_flex" version = "0.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7ef0d4ed8669f8f8826eb00dc878084aa8f253506c4fd5e8f58f5bce72ddb97e" dependencies = [ "twox-hash", ] [[package]] name = "md-5" version = "0.10.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d89e7ee0cfbedfc4da3340218492196241d89eefb6dab27de5df917a6d2e78cf" dependencies = [ "cfg-if", "digest 0.10.7", ] [[package]] name = "md-5" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" dependencies = [ "cfg-if", "digest 0.11.3", ] [[package]] name = "memchr" version = "2.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] name = "mimalloc" version = "0.1.52" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2d4139bb28d14ad1facf21d5eb8825051b326e172d216b39f6d31df53cc97862" dependencies = [ "libmimalloc-sys", ] [[package]] name = "miniz_oxide" version = "0.8.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1fa76a2c86f704bdb222d66965fb3d63269ce38518b83cb0575fca855ebb6316" dependencies = [ "adler2", "simd-adler32", ] [[package]] name = "mio" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "02bd0af71c67b473010cbbc60715ee815645a4dc942899111f494b4b737d6fda" dependencies = [ "libc", "wasi", "windows-sys 0.61.2", ] [[package]] name = "multimap" version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1d87ecb2933e8aeadb3e3a02b828fed80a7528047e68b4f424523a0981a3a084" [[package]] name = "num-bigint" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a5e44f723f1133c9deac646763579fdb3ac745e418f2a7af9cd0c431da1f20b9" dependencies = [ "num-integer", "num-traits", ] [[package]] name = "num-complex" version = "0.4.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "73f88a1307638156682bada9d7604135552957b7818057dcef22705b4d509495" dependencies = [ "num-traits", ] [[package]] name = "num-integer" version = "0.1.46" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7969661fd2958a5cb096e56c8e1ad0444ac2bbcd0061bd28660485a44879858f" dependencies = [ "num-traits", ] [[package]] name = "num-traits" version = "0.2.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "071dfc062690e90b734c0b2273ce72ad0ffa95f0c74596bc250dcfd960262841" dependencies = [ "autocfg", "libm", ] [[package]] name = "object" version = "0.37.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff76201f031d8863c38aa7f905eca4f53abbfa15f609db4277d44cd8938f33fe" dependencies = [ "memchr", ] [[package]] name = "object_store" version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "622acbc9100d3c10e2ee15804b0caa40e55c933d5aa53814cd520805b7958a49" dependencies = [ "async-trait", "base64", "bytes", "chrono", "form_urlencoded", "futures-channel", "futures-core", "futures-util", "http", "http-body-util", "httparse", "humantime", "hyper", "itertools", "md-5 0.10.6", "parking_lot", "percent-encoding", "quick-xml", "rand 0.10.1", "reqwest", "ring", "rustls-pki-types", "serde", "serde_json", "serde_urlencoded", "thiserror", "tokio", "tracing", "url", "walkdir", "wasm-bindgen-futures", "web-time", ] [[package]] name = "once_cell" version = "1.21.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50" [[package]] name = "openssl-probe" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe" [[package]] name = "ordered-float" version = "2.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "68f19d67e5a2795c94e73e0bb1cc1a7edeb2e28efd39e2e1c9b7a40c1108b11c" dependencies = [ "num-traits", ] [[package]] name = "parking_lot" version = "0.12.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93857453250e3077bd71ff98b6a65ea6621a19bb0f559a85248955ac12c45a1a" dependencies = [ "lock_api", "parking_lot_core", ] [[package]] name = "parking_lot_core" version = "0.9.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2621685985a2ebf1c516881c026032ac7deafcda1a2c9b7850dc81e3dfcb64c1" dependencies = [ "cfg-if", "libc", "redox_syscall", "smallvec", "windows-link", ] [[package]] name = "parquet" version = "58.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5dafa7d01085b62a47dd0c1829550a0a36710ea9c4fe358a05a85477cec8a908" dependencies = [ "ahash", "arrow-array", "arrow-buffer", "arrow-data", "arrow-ipc", "arrow-schema", "arrow-select", "base64", "brotli", "bytes", "chrono", "flate2", "futures", "half", "hashbrown 0.17.1", "lz4_flex", "num-bigint", "num-integer", "num-traits", "object_store", "paste", "seq-macro", "simdutf8", "snap", "thrift", "tokio", "twox-hash", "zstd", ] [[package]] name = "paste" version = "1.0.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "57c0d7b74b563b49d38dae00a0c37d4d6de9b432382b2892f0574ddcae73fd0a" [[package]] name = "pbjson" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "898bac3fa00d0ba57a4e8289837e965baa2dee8c3749f3b11d45a64b4223d9c3" dependencies = [ "base64", "serde", ] [[package]] name = "pbjson-build" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "af22d08a625a2213a78dbb0ffa253318c5c79ce3133d32d296655a7bdfb02095" dependencies = [ "heck", "itertools", "prost", "prost-types", ] [[package]] name = "pbjson-types" version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e748e28374f10a330ee3bb9f29b828c0ac79831a32bab65015ad9b661ead526" dependencies = [ "bytes", "chrono", "pbjson", "pbjson-build", "prost", "prost-build", "serde", ] [[package]] name = "percent-encoding" version = "2.3.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220" [[package]] name = "petgraph" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8701b58ea97060d5e5b155d383a69952a60943f0e6dfe30b04c287beb0b27455" dependencies = [ "fixedbitset", "hashbrown 0.15.5", "indexmap", "serde", ] [[package]] name = "phf" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "913273894cec178f401a31ec4b656318d95473527be05c0752cc41cdc32be8b7" dependencies = [ "phf_shared", ] [[package]] name = "phf_shared" version = "0.12.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "06005508882fb681fd97892ecff4b7fd0fee13ef1aa569f8695dae7ab9099981" dependencies = [ "siphasher", ] [[package]] name = "pin-project" version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2466b2336ed02bcdca6b294417127b90ec92038d1d5c4fbeac971a922e0e0924" dependencies = [ "pin-project-internal", ] [[package]] name = "pin-project-internal" version = "1.1.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "pin-project-lite" version = "0.2.17" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd" [[package]] name = "pkg-config" version = "0.3.33" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e" [[package]] name = "portable-atomic" version = "1.13.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c33a9471896f1c69cecef8d20cbe2f7accd12527ce60845ff44c153bb2a21b49" [[package]] name = "potential_utf" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564" dependencies = [ "zerovec", ] [[package]] name = "ppv-lite86" version = "0.2.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9" dependencies = [ "zerocopy", ] [[package]] name = "prettyplease" version = "0.2.37" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", "syn", ] [[package]] name = "proc-macro-crate" version = "3.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e67ba7e9b2b56446f1d419b1d807906278ffa1a658a8a5d8a39dcb1f5a78614f" dependencies = [ "toml_edit", ] [[package]] name = "proc-macro2" version = "1.0.106" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934" dependencies = [ "unicode-ident", ] [[package]] name = "prost" version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "528ac67416ff8646872a3c02cad9cc4ee5dc9f9540c9b10771855c95cb2e5ae1" dependencies = [ "bytes", "prost-derive", ] [[package]] name = "prost-build" version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042" dependencies = [ "heck", "itertools", "log", "multimap", "petgraph", "prettyplease", "prost", "prost-types", "regex", "syn", "tempfile", ] [[package]] name = "prost-derive" version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", "itertools", "proc-macro2", "quote", "syn", ] [[package]] name = "prost-types" version = "0.14.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f94967dc7688f3054c7fac87473ffae4cc4c3904800e2d9f5b857246d8963b0a" dependencies = [ "prost", ] [[package]] name = "protobuf-src" version = "2.1.1+27.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6217c3504da19b85a3a4b2e9a5183d635822d83507ba0986624b5c05b83bfc40" dependencies = [ "cmake", ] [[package]] name = "psm" version = "0.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "645dbe486e346d9b5de3ef16ede18c26e6c70ad97418f4874b8b1889d6e761ea" dependencies = [ "ar_archive_writer", "cc", ] [[package]] name = "pyo3" version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91fd8e38a3b50ed1167fb981cd6fd60147e091784c427b8f7183a7ee32c31c12" dependencies = [ "libc", "once_cell", "portable-atomic", "pyo3-build-config", "pyo3-ffi", "pyo3-macros", ] [[package]] name = "pyo3-async-runtimes" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9e7364a95bf00e8377bbf9b0f09d7ff9715a29d8fcf93b47d1a967363b973178" dependencies = [ "futures-channel", "futures-util", "once_cell", "pin-project-lite", "pyo3", "tokio", ] [[package]] name = "pyo3-build-config" version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e368e7ddfdeb98c9bca7f8383be1648fd84ab466bf2bc015e94008db6d35611e" dependencies = [ "python3-dll-a", "target-lexicon", ] [[package]] name = "pyo3-ffi" version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7f29e10af80b1f7ccaf7f69eace800a03ecd13e883acfacc1e5d0988605f651e" dependencies = [ "libc", "pyo3-build-config", ] [[package]] name = "pyo3-log" version = "0.13.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f64083bd3a16a353d9d62335808e8e13d0552d2a2b83fdb084496192dcfa9fcd" dependencies = [ "arc-swap", "log", "pyo3", ] [[package]] name = "pyo3-macros" version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "df6e520eff47c45997d2fc7dd8214b25dd1310918bbb2642156ef66a67f29813" dependencies = [ "proc-macro2", "pyo3-macros-backend", "quote", "syn", ] [[package]] name = "pyo3-macros-backend" version = "0.28.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c4cdc218d835738f81c2338f822078af45b4afdf8b2e33cbb5916f108b813acb" dependencies = [ "heck", "proc-macro2", "pyo3-build-config", "quote", "syn", ] [[package]] name = "python3-dll-a" version = "0.2.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d80ba7540edb18890d444c5aa8e1f1f99b1bdf26fb26ae383135325f4a36042b" dependencies = [ "cc", ] [[package]] name = "quick-xml" version = "0.39.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cdcc8dd4e2f670d309a5f0e83fe36dfdc05af317008fea29144da1a2ac858e5e" dependencies = [ "memchr", "serde", ] [[package]] name = "quinn" version = "0.11.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" dependencies = [ "bytes", "cfg_aliases", "pin-project-lite", "quinn-proto", "quinn-udp", "rustc-hash", "rustls", "socket2", "thiserror", "tokio", "tracing", "web-time", ] [[package]] name = "quinn-proto" version = "0.11.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" dependencies = [ "bytes", "getrandom 0.3.4", "lru-slab", "rand 0.9.4", "ring", "rustc-hash", "rustls", "rustls-pki-types", "slab", "thiserror", "tinyvec", "tracing", "web-time", ] [[package]] name = "quinn-udp" version = "0.5.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd" dependencies = [ "cfg_aliases", "libc", "once_cell", "socket2", "tracing", "windows-sys 0.60.2", ] [[package]] name = "quote" version = "1.0.45" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" dependencies = [ "proc-macro2", ] [[package]] name = "r-efi" version = "5.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f" [[package]] name = "r-efi" version = "6.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dcc9c7d52a811697d2151c701e0d08956f92b0e24136cf4cf27b57a6a0d9bf" [[package]] name = "rand" version = "0.9.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "44c5af06bb1b7d3216d91932aed5265164bf384dc89cd6ba05cf59a35f5f76ea" dependencies = [ "rand_chacha", "rand_core 0.9.5", ] [[package]] name = "rand" version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" dependencies = [ "chacha20", "getrandom 0.4.3", "rand_core 0.10.1", ] [[package]] name = "rand_chacha" version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb" dependencies = [ "ppv-lite86", "rand_core 0.9.5", ] [[package]] name = "rand_core" version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c" dependencies = [ "getrandom 0.3.4", ] [[package]] name = "rand_core" version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" [[package]] name = "recursive" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0786a43debb760f491b1bc0269fe5e84155353c67482b9e60d0cfb596054b43e" dependencies = [ "recursive-proc-macro-impl", "stacker", ] [[package]] name = "recursive-proc-macro-impl" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", "syn", ] [[package]] name = "redox_syscall" version = "0.5.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ed2bf2547551a7053d6fdfafda3f938979645c44812fbfcda098faae3f1a362d" dependencies = [ "bitflags", ] [[package]] name = "regex" version = "1.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba" dependencies = [ "aho-corasick", "memchr", "regex-automata", "regex-syntax", ] [[package]] name = "regex-automata" version = "0.4.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f" dependencies = [ "aho-corasick", "memchr", "regex-syntax", ] [[package]] name = "regex-syntax" version = "0.8.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6f6ff9a378485b298a5286656da665ba74413d36db0979633275d2e708145d4" [[package]] name = "regress" version = "0.10.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2057b2325e68a893284d1538021ab90279adac1139957ca2a74426c6f118fb48" dependencies = [ "hashbrown 0.16.1", "memchr", ] [[package]] name = "reqwest" version = "0.12.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "eddd3ca559203180a307f12d114c268abf583f59b03cb906fd0b3ff8646c1147" dependencies = [ "base64", "bytes", "futures-core", "futures-util", "h2", "http", "http-body", "http-body-util", "hyper", "hyper-rustls", "hyper-util", "js-sys", "log", "percent-encoding", "pin-project-lite", "quinn", "rustls", "rustls-native-certs", "rustls-pki-types", "serde", "serde_json", "serde_urlencoded", "sync_wrapper", "tokio", "tokio-rustls", "tokio-util", "tower", "tower-http", "tower-service", "url", "wasm-bindgen", "wasm-bindgen-futures", "wasm-streams", "web-sys", ] [[package]] name = "ring" version = "0.17.14" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7" dependencies = [ "cc", "cfg-if", "getrandom 0.2.17", "libc", "untrusted", "windows-sys 0.52.0", ] [[package]] name = "rustc-hash" version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" [[package]] name = "rustc_version" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cfcb3a22ef46e85b45de6ee7e79d063319ebb6594faafcf1c225ea92ab6e9b92" dependencies = [ "semver", ] [[package]] name = "rustix" version = "1.1.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190" dependencies = [ "bitflags", "errno", "libc", "linux-raw-sys", "windows-sys 0.61.2", ] [[package]] name = "rustls" version = "0.23.40" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ef86cd5876211988985292b91c96a8f2d298df24e75989a43a3c73f2d4d8168b" dependencies = [ "once_cell", "ring", "rustls-pki-types", "rustls-webpki", "subtle", "zeroize", ] [[package]] name = "rustls-native-certs" version = "0.8.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "dab5152771c58876a2146916e53e35057e1a4dfa2b9df0f0305b07f611fdea4d" dependencies = [ "openssl-probe", "rustls-pki-types", "schannel", "security-framework", ] [[package]] name = "rustls-pki-types" version = "1.14.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "30a7197ae7eb376e574fe940d068c30fe0462554a3ddbe4eca7838e049c937a9" dependencies = [ "web-time", "zeroize", ] [[package]] name = "rustls-webpki" version = "0.103.13" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "61c429a8649f110dddef65e2a5ad240f747e85f7758a6bccc7e5777bd33f756e" dependencies = [ "ring", "rustls-pki-types", "untrusted", ] [[package]] name = "rustversion" version = "1.0.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d" [[package]] name = "ryu" version = "1.0.23" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f" [[package]] name = "same-file" version = "1.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502" dependencies = [ "winapi-util", ] [[package]] name = "schannel" version = "0.1.29" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91c1b7e4904c873ef0710c1f407dde2e6287de2bebc1bbbf7d430bb7cbffd939" dependencies = [ "windows-sys 0.61.2", ] [[package]] name = "schemars" version = "0.8.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3fbf2ae1b8bc8e02df939598064d22402220cd5bbcca1c76f7d6a310974d5615" dependencies = [ "dyn-clone", "schemars_derive", "serde", "serde_json", ] [[package]] name = "schemars_derive" version = "0.8.22" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32e265784ad618884abaea0600a9adf15393368d840e0222d101a072f3f7534d" dependencies = [ "proc-macro2", "quote", "serde_derive_internals", "syn", ] [[package]] name = "scopeguard" version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49" [[package]] name = "security-framework" version = "3.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7f4bc775c73d9a02cde8bf7b2ec4c9d12743edf609006c7facc23998404cd1d" dependencies = [ "bitflags", "core-foundation", "core-foundation-sys", "libc", "security-framework-sys", ] [[package]] name = "security-framework-sys" version = "2.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ce2691df843ecc5d231c0b14ece2acc3efb62c0a398c7e1d875f3983ce020e3" dependencies = [ "core-foundation-sys", "libc", ] [[package]] name = "semver" version = "1.0.28" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" dependencies = [ "serde", "serde_core", ] [[package]] name = "seq-macro" version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" [[package]] name = "serde" version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" dependencies = [ "serde_core", "serde_derive", ] [[package]] name = "serde_core" version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "serde_derive_internals" version = "0.29.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "serde_json" version = "1.0.150" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e8014e44b4736ed0538adeecded0fce2a272f22dc9578a7eb6b2d9993c74cfb9" dependencies = [ "indexmap", "itoa", "memchr", "serde", "serde_core", "zmij", ] [[package]] name = "serde_tokenstream" version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d7c49585c52c01f13c5c2ebb333f14f6885d76daa768d8a037d28017ec538c69" dependencies = [ "proc-macro2", "quote", "serde", "syn", ] [[package]] name = "serde_urlencoded" version = "0.7.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d3491c14715ca2294c4d6a88f15e84739788c1d030eed8c110436aafdaa2f3fd" dependencies = [ "form_urlencoded", "itoa", "ryu", "serde", ] [[package]] name = "serde_yaml" version = "0.9.34+deprecated" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" dependencies = [ "indexmap", "itoa", "ryu", "serde", "unsafe-libyaml", ] [[package]] name = "sha1" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "aacc4cc499359472b4abe1bf11d0b12e688af9a805fa5e3016f9a386dc2d0214" dependencies = [ "cfg-if", "cpufeatures", "digest 0.11.3", ] [[package]] name = "sha2" version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" dependencies = [ "cfg-if", "cpufeatures", "digest 0.11.3", ] [[package]] name = "sha2-const-stable" version = "0.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5f179d4e11094a893b82fff208f74d448a7512f99f5a0acbd5c679b705f83ed9" [[package]] name = "shlex" version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8fadd59c855ef2080decdef8ff161eb6661b86933c9d82e5ba29dc602a55aba" [[package]] name = "simd-adler32" version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" [[package]] name = "simdutf8" version = "0.1.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" [[package]] name = "siphasher" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ee5873ec9cce0195efcb7a4e9507a04cd49aec9c83d0389df45b1ef7ba2e649" [[package]] name = "slab" version = "0.4.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5" [[package]] name = "smallvec" version = "1.15.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ed6a63f02c8539c91a8685a86f4099661ba3da017932f6ebbea6de3f0fa7c90" [[package]] name = "snap" version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b6b67fb9a61334225b5b790716f609cd58395f895b3fe8b328786812a40bc3b" [[package]] name = "socket2" version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "52d1cfed4120b4d927bf7c0f86d2087a4a7d6027c906d9f9d525a80573b9be51" dependencies = [ "libc", "windows-sys 0.61.2", ] [[package]] name = "sqlparser" version = "0.62.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13c6d1b651dc4edf07eead2a0c6c78016ce971bc2c10da5266861b13f25e7cec" dependencies = [ "log", "recursive", "sqlparser_derive", ] [[package]] name = "sqlparser_derive" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6dd45d8fc1c79299bfbb7190e42ccbbdf6a5f52e4a6ad98d92357ea965bd289" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "stabby" version = "72.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7b834ec7ced12095fea1e4b07dcb7e8cf2b59b18afa3eac52494d835965a5ec" dependencies = [ "rustversion", "stabby-abi", ] [[package]] name = "stabby-abi" version = "72.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff1a4f477858a5bdf927c9fab7f579899de9b13e39f8b3b3b300c89fbab632f4" dependencies = [ "rustc_version", "rustversion", "sha2-const-stable", "stabby-macros", ] [[package]] name = "stabby-macros" version = "72.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b31c4b2434980b67ad83f300a58088ba14d59454dcd79ba3d87419bbd924d31e" dependencies = [ "proc-macro-crate", "proc-macro2", "quote", "syn", ] [[package]] name = "stable_deref_trait" version = "1.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6ce2be8dc25455e1f91df71bfa12ad37d7af1092ae736f3a6cd0e37bc7810596" [[package]] name = "stacker" version = "0.1.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "640c8cdd92b6b12f5bcb1803ca3bbf5ab96e5e6b6b96b9ab77dabe9e880b3190" dependencies = [ "cc", "cfg-if", "libc", "psm", "windows-sys 0.61.2", ] [[package]] name = "strum_macros" version = "0.28.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab85eea0270ee17587ed4156089e10b9e6880ee688791d45a905f5b1ca36f664" dependencies = [ "heck", "proc-macro2", "quote", "syn", ] [[package]] name = "substrait" version = "0.63.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e620ff4d5c02fd6f7752931aa74b16a26af66a63022cc1ad412c77edbe0bab47" dependencies = [ "heck", "indexmap", "pbjson", "pbjson-build", "pbjson-types", "prettyplease", "prost", "prost-build", "prost-types", "protobuf-src", "regress", "schemars", "semver", "serde", "serde_json", "serde_yaml", "syn", "typify", "walkdir", ] [[package]] name = "subtle" version = "2.6.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "13c2bddecc57b384dee18652358fb23172facb8a2c51ccc10d74c157bdea3292" [[package]] name = "syn" version = "2.0.118" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" dependencies = [ "proc-macro2", "quote", "unicode-ident", ] [[package]] name = "sync_wrapper" version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263" dependencies = [ "futures-core", ] [[package]] name = "synstructure" version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "target-lexicon" version = "0.13.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca" [[package]] name = "tempfile" version = "3.27.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", "getrandom 0.4.3", "once_cell", "rustix", "windows-sys 0.61.2", ] [[package]] name = "thiserror" version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4" dependencies = [ "thiserror-impl", ] [[package]] name = "thiserror-impl" version = "2.0.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "thrift" version = "0.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" dependencies = [ "byteorder", "integer-encoding", "ordered-float", ] [[package]] name = "tiny-keccak" version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "2c9d3793400a45f954c52e73d068316d76b6f4e36977e3fcebb13a2721e80237" dependencies = [ "crunchy", ] [[package]] name = "tinystr" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d" dependencies = [ "displaydoc", "zerovec", ] [[package]] name = "tinyvec" version = "1.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3" dependencies = [ "tinyvec_macros", ] [[package]] name = "tinyvec_macros" version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1f3ccbac311fea05f86f61904b462b55fb3df8837a366dfc601a0161d0532f20" [[package]] name = "tokio" version = "1.52.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8fc7f01b389ac15039e4dc9531aa973a135d7a4135281b12d7c1bc79fd57fffe" dependencies = [ "bytes", "libc", "mio", "pin-project-lite", "socket2", "tokio-macros", "windows-sys 0.61.2", ] [[package]] name = "tokio-macros" version = "2.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "385a6cb71ab9ab790c5fe8d67f1645e6c450a7ce006a33de03daa956cf70a496" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "tokio-rustls" version = "0.26.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61" dependencies = [ "rustls", "tokio", ] [[package]] name = "tokio-stream" version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32da49809aab5c3bc678af03902d4ccddea2a87d028d86392a4b1560c6906c70" dependencies = [ "futures-core", "pin-project-lite", "tokio", "tokio-util", ] [[package]] name = "tokio-util" version = "0.7.18" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ae9cec805b01e8fc3fd2fe289f89149a9b66dd16786abd8b19cfa7b48cb0098" dependencies = [ "bytes", "futures-core", "futures-sink", "pin-project-lite", "tokio", ] [[package]] name = "toml_datetime" version = "1.1.1+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3165f65f62e28e0115a00b2ebdd37eb6f3b641855f9d636d3cd4103767159ad7" dependencies = [ "serde_core", ] [[package]] name = "toml_edit" version = "0.25.12+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d2153edc6955a6c354fad8f5efd38b6a8769bdccf9fe50f8e1329f81b0baa5d7" dependencies = [ "indexmap", "toml_datetime", "toml_parser", "winnow", ] [[package]] name = "toml_parser" version = "1.1.2+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a2abe9b86193656635d2411dc43050282ca48aa31c2451210f4202550afb7526" dependencies = [ "winnow", ] [[package]] name = "tower" version = "0.5.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" dependencies = [ "futures-core", "futures-util", "pin-project-lite", "sync_wrapper", "tokio", "tower-layer", "tower-service", ] [[package]] name = "tower-http" version = "0.6.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4cfcf7e2740e6fc6d4d688b4ef00650406bb94adf4731e43c096c3a19fe40840" dependencies = [ "bitflags", "bytes", "futures-util", "http", "http-body", "pin-project-lite", "tower", "tower-layer", "tower-service", "url", ] [[package]] name = "tower-layer" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "121c2a6cda46980bb0fcd1647ffaf6cd3fc79a013de288782836f6df9c48780e" [[package]] name = "tower-service" version = "0.3.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8df9b6e13f2d32c91b9bd719c00d1958837bc7dec474d94952798cc8e69eeec3" [[package]] name = "tracing" version = "0.1.44" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100" dependencies = [ "pin-project-lite", "tracing-attributes", "tracing-core", ] [[package]] name = "tracing-attributes" version = "0.1.31" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "tracing-core" version = "0.1.36" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a" dependencies = [ "once_cell", ] [[package]] name = "try-lock" version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" [[package]] name = "twox-hash" version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ea3136b675547379c4bd395ca6b938e5ad3c3d20fad76e7fe85f9e0d011419c" dependencies = [ "rand 0.9.4", ] [[package]] name = "typenum" version = "1.20.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6f5e870be6c3b371b77fe0ee0bafb859fa4964b4404c27de1d380043c4dda20" [[package]] name = "typify" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6d5bcc6f62eb1fa8aa4098f39b29f93dcb914e17158b76c50360911257aa629" dependencies = [ "typify-impl", "typify-macro", ] [[package]] name = "typify-impl" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a1eb359f7ffa4f9ebe947fa11a1b2da054564502968db5f317b7e37693cb2240" dependencies = [ "heck", "log", "proc-macro2", "quote", "regress", "schemars", "semver", "serde", "serde_json", "syn", "thiserror", "unicode-ident", ] [[package]] name = "typify-macro" version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "911c32f3c8514b048c1b228361bebb5e6d73aeec01696e8cc0e82e2ffef8ab7a" dependencies = [ "proc-macro2", "quote", "schemars", "semver", "serde", "serde_json", "serde_tokenstream", "syn", "typify-impl", ] [[package]] name = "unicode-ident" version = "1.0.24" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75" [[package]] name = "unicode-segmentation" version = "1.13.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c6f5d3c3b1bf09027a88a6bc961fc00497d651009560b5463668dc81b0fa87a8" [[package]] name = "unicode-width" version = "0.2.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254" [[package]] name = "unsafe-libyaml" version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" [[package]] name = "untrusted" version = "0.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1" [[package]] name = "url" version = "2.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ff67a8a4397373c3ef660812acab3268222035010ab8680ec4215f38ba3d0eed" dependencies = [ "form_urlencoded", "idna", "percent-encoding", "serde", ] [[package]] name = "utf8_iter" version = "1.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b6c140620e7ffbb22c2dee59cafe6084a59b5ffc27a8859a5f0d494b5d52b6be" [[package]] name = "uuid" version = "1.23.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" dependencies = [ "getrandom 0.4.3", "js-sys", "wasm-bindgen", ] [[package]] name = "version_check" version = "0.9.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a" [[package]] name = "walkdir" version = "2.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b" dependencies = [ "same-file", "winapi-util", ] [[package]] name = "want" version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e" dependencies = [ "try-lock", ] [[package]] name = "wasi" version = "0.11.1+wasi-snapshot-preview1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" version = "1.0.4+wasi-0.2.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" dependencies = [ "wit-bindgen", ] [[package]] name = "wasm-bindgen" version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8ddb3f79143bced6de84270411622a2699cee572fc0875aeaf1e7867cf9fca1a" dependencies = [ "cfg-if", "once_cell", "rustversion", "wasm-bindgen-macro", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-futures" version = "0.4.75" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "503b14d284f2c8dac03b819967e155ea753f573586193b2b2c95990cb5d69280" dependencies = [ "js-sys", "wasm-bindgen", ] [[package]] name = "wasm-bindgen-macro" version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4e21a184b13fb19e157296e2c46056aec9092264fab83e4ba59e68c61b323c3d" dependencies = [ "quote", "wasm-bindgen-macro-support", ] [[package]] name = "wasm-bindgen-macro-support" version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fecefd9c35bd935a20fc3fc344b5f29138961e4f47fb03297d88f2587afb5ebd" dependencies = [ "bumpalo", "proc-macro2", "quote", "syn", "wasm-bindgen-shared", ] [[package]] name = "wasm-bindgen-shared" version = "0.2.125" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "23939e44bb9a5d7576fa2b563dc2e136628f1224e88a8deed09e04858b77871f" dependencies = [ "unicode-ident", ] [[package]] name = "wasm-streams" version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "15053d8d85c7eccdbefef60f06769760a563c7f0a9d6902a13d35c7800b0ad65" dependencies = [ "futures-util", "js-sys", "wasm-bindgen", "wasm-bindgen-futures", "web-sys", ] [[package]] name = "web-sys" version = "0.3.102" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a6430a72df5eb332242960fe84b3002a241163998241eb596d4f739b9757061d" dependencies = [ "js-sys", "wasm-bindgen", ] [[package]] name = "web-time" version = "1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb" dependencies = [ "js-sys", "wasm-bindgen", ] [[package]] name = "winapi-util" version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ "windows-sys 0.61.2", ] [[package]] name = "windows-core" version = "0.62.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8e83a14d34d0623b51dce9581199302a221863196a1dde71a7663a4c2be9deb" dependencies = [ "windows-implement", "windows-interface", "windows-link", "windows-result", "windows-strings", ] [[package]] name = "windows-implement" version = "0.60.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "windows-interface" version = "0.59.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "windows-link" version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5" [[package]] name = "windows-result" version = "0.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7781fa89eaf60850ac3d2da7af8e5242a5ea78d1a11c49bf2910bb5a73853eb5" dependencies = [ "windows-link", ] [[package]] name = "windows-strings" version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7837d08f69c77cf6b07689544538e017c1bfcf57e34b4c0ff58e6c2cd3b37091" dependencies = [ "windows-link", ] [[package]] name = "windows-sys" version = "0.52.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "282be5f36a8ce781fad8c8ae18fa3f9beff57ec1b52cb3de0789201425d9a33d" dependencies = [ "windows-targets 0.52.6", ] [[package]] name = "windows-sys" version = "0.60.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f2f500e4d28234f72040990ec9d39e3a6b950f9f22d3dba18416c35882612bcb" dependencies = [ "windows-targets 0.53.5", ] [[package]] name = "windows-sys" version = "0.61.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc" dependencies = [ "windows-link", ] [[package]] name = "windows-targets" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9b724f72796e036ab90c1021d4780d4d3d648aca59e491e6b98e725b84e99973" dependencies = [ "windows_aarch64_gnullvm 0.52.6", "windows_aarch64_msvc 0.52.6", "windows_i686_gnu 0.52.6", "windows_i686_gnullvm 0.52.6", "windows_i686_msvc 0.52.6", "windows_x86_64_gnu 0.52.6", "windows_x86_64_gnullvm 0.52.6", "windows_x86_64_msvc 0.52.6", ] [[package]] name = "windows-targets" version = "0.53.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4945f9f551b88e0d65f3db0bc25c33b8acea4d9e41163edf90dcd0b19f9069f3" dependencies = [ "windows-link", "windows_aarch64_gnullvm 0.53.1", "windows_aarch64_msvc 0.53.1", "windows_i686_gnu 0.53.1", "windows_i686_gnullvm 0.53.1", "windows_i686_msvc 0.53.1", "windows_x86_64_gnu 0.53.1", "windows_x86_64_gnullvm 0.53.1", "windows_x86_64_msvc 0.53.1", ] [[package]] name = "windows_aarch64_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32a4622180e7a0ec044bb555404c800bc9fd9ec262ec147edd5989ccd0c02cd3" [[package]] name = "windows_aarch64_gnullvm" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a9d8416fa8b42f5c947f8482c43e7d89e73a173cead56d044f6a56104a6d1b53" [[package]] name = "windows_aarch64_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "09ec2a7bb152e2252b53fa7803150007879548bc709c039df7627cabbd05d469" [[package]] name = "windows_aarch64_msvc" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b9d782e804c2f632e395708e99a94275910eb9100b2114651e04744e9b125006" [[package]] name = "windows_i686_gnu" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8e9b5ad5ab802e97eb8e295ac6720e509ee4c243f69d781394014ebfe8bbfa0b" [[package]] name = "windows_i686_gnu" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "960e6da069d81e09becb0ca57a65220ddff016ff2d6af6a223cf372a506593a3" [[package]] name = "windows_i686_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0eee52d38c090b3caa76c563b86c3a4bd71ef1a819287c19d586d7334ae8ed66" [[package]] name = "windows_i686_gnullvm" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fa7359d10048f68ab8b09fa71c3daccfb0e9b559aed648a8f95469c27057180c" [[package]] name = "windows_i686_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "240948bc05c5e7c6dabba28bf89d89ffce3e303022809e73deaefe4f6ec56c66" [[package]] name = "windows_i686_msvc" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e7ac75179f18232fe9c285163565a57ef8d3c89254a30685b57d83a38d326c2" [[package]] name = "windows_x86_64_gnu" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "147a5c80aabfbf0c7d901cb5895d1de30ef2907eb21fbbab29ca94c5b08b1a78" [[package]] name = "windows_x86_64_gnu" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9c3842cdd74a865a8066ab39c8a7a473c0778a3f29370b5fd6b4b9aa7df4a499" [[package]] name = "windows_x86_64_gnullvm" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "24d5b23dc417412679681396f2b49f3de8c1473deb516bd34410872eff51ed0d" [[package]] name = "windows_x86_64_gnullvm" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ffa179e2d07eee8ad8f57493436566c7cc30ac536a3379fdf008f47f6bb7ae1" [[package]] name = "windows_x86_64_msvc" version = "0.52.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "589f6da84c646204747d1270a2a5661ea66ed1cced2631d546fdfb155959f9ec" [[package]] name = "windows_x86_64_msvc" version = "0.53.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d6bbff5f0aada427a1e5a6da5f1f98158182f26556f345ac9e04d36d0ebed650" [[package]] name = "winnow" version = "1.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0592e1c9d151f854e6fd382574c3a0855250e1d9b2f99d9281c6e6391af352f1" dependencies = [ "memchr", ] [[package]] name = "wit-bindgen" version = "0.57.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" [[package]] name = "writeable" version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4" [[package]] name = "yoke" version = "0.8.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "709fe23a0424b6a435d82152b1bd3fdfb0833487d5fa90d05d42762a9891fef5" dependencies = [ "stable_deref_trait", "yoke-derive", "zerofrom", ] [[package]] name = "yoke-derive" version = "0.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" dependencies = [ "proc-macro2", "quote", "syn", "synstructure", ] [[package]] name = "zerocopy" version = "0.8.52" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ce1022995ff5ff5d841ad7d994facc23098cd40152f2c1d11cd607c6f530653f" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" version = "0.8.52" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "zerofrom" version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0ec05a11813ea801ff6d75110ad09cd0824ddba17dfe17128ea0d5f68e6c5272" dependencies = [ "zerofrom-derive", ] [[package]] name = "zerofrom-derive" version = "0.1.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" dependencies = [ "proc-macro2", "quote", "syn", "synstructure", ] [[package]] name = "zeroize" version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" [[package]] name = "zerotrie" version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf" dependencies = [ "displaydoc", "yoke", "zerofrom", ] [[package]] name = "zerovec" version = "0.11.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239" dependencies = [ "yoke", "zerofrom", "zerovec-derive", ] [[package]] name = "zerovec-derive" version = "0.11.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" dependencies = [ "proc-macro2", "quote", "syn", ] [[package]] name = "zlib-rs" version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" [[package]] name = "zmij" version = "1.0.21" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" [[package]] name = "zstd" version = "0.13.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e91ee311a569c327171651566e07972200e76fcfe2242a4fa446149a3881c08a" dependencies = [ "zstd-safe", ] [[package]] name = "zstd-safe" version = "7.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8f49c4d5f0abb602a93fb8736af2a4f4dd9512e36f7f570d66e65ff867ed3b9d" dependencies = [ "zstd-sys", ] [[package]] name = "zstd-sys" version = "2.0.16+zstd.1.5.7" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "91e19ebc2adc8f83e43039e79776e3fda8ca919132d68a1fed6a5faca2683748" dependencies = [ "cc", "pkg-config", ] apache-datafusion-python-54.0.0/Cargo.toml000066400000000000000000000052461521727353300204420ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. [workspace.package] version = "54.0.0" homepage = "https://datafusion.apache.org/python" repository = "https://github.com/apache/datafusion-python" authors = ["Apache DataFusion "] description = "Apache DataFusion DataFrame and SQL Query Engine" readme = "README.md" license = "Apache-2.0" edition = "2024" rust-version = "1.88" [workspace] members = ["crates/core", "crates/util", "examples/datafusion-ffi-example"] resolver = "3" [workspace.dependencies] tokio = { version = "1.52" } pyo3 = { version = "0.28" } pyo3-async-runtimes = { version = "0.28" } pyo3-log = "0.13.3" chrono = { version = "0.4", default-features = false } arrow = { version = "58" } arrow-array = { version = "58" } arrow-schema = { version = "58" } arrow-select = { version = "58" } datafusion = { version = "54" } datafusion-substrait = { version = "54" } datafusion-proto = { version = "54" } datafusion-ffi = { version = "54" } datafusion-catalog = { version = "54", default-features = false } datafusion-common = { version = "54", default-features = false } datafusion-functions-aggregate = { version = "54" } datafusion-functions-window = { version = "54" } datafusion-spark = { version = "54" } datafusion-expr = { version = "54" } prost = "0.14.3" serde_json = "1" uuid = { version = "1.23" } mimalloc = { version = "0.1", default-features = false } async-trait = "0.1.89" futures = "0.3" cstr = "0.2" object_store = { version = "0.13.1" } url = "2" log = "0.4.29" parking_lot = "0.12" prost-types = "0.14.3" # keep in line with `datafusion-substrait` pyo3-build-config = "0.28" datafusion-python-util = { path = "crates/util", version = "54.0.0" } [profile.release] lto = "thin" codegen-units = 2 # We cannot publish to crates.io with any patches in the below section. Developers # must remove any entries in this section before creating a release candidate. [patch.crates-io] apache-datafusion-python-54.0.0/LICENSE.txt000066400000000000000000000261361521727353300203360ustar00rootroot00000000000000 Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. apache-datafusion-python-54.0.0/README.md000066400000000000000000000307251521727353300177710ustar00rootroot00000000000000 # DataFusion in Python [![Python test](https://github.com/apache/datafusion-python/actions/workflows/test.yaml/badge.svg)](https://github.com/apache/datafusion-python/actions/workflows/test.yaml) [![Python Release Build](https://github.com/apache/datafusion-python/actions/workflows/build.yml/badge.svg)](https://github.com/apache/datafusion-python/actions/workflows/build.yml) This is a Python library that binds to [Apache Arrow](https://arrow.apache.org/) in-memory query engine [DataFusion](https://github.com/apache/datafusion). DataFusion's Python bindings can be used as a foundation for building new data systems in Python. Here are some examples: - [Dask SQL](https://github.com/dask-contrib/dask-sql) uses DataFusion's Python bindings for SQL parsing, query planning, and logical plan optimizations, and then transpiles the logical plan to Dask operations for execution. - [DataFusion Ballista](https://github.com/apache/datafusion-ballista) is a distributed SQL query engine that extends DataFusion's Python bindings for distributed use cases. - [DataFusion Ray](https://github.com/apache/datafusion-ray) is another distributed query engine that uses DataFusion's Python bindings. ## Features - Execute queries using SQL or DataFrames against CSV, Parquet, and JSON data sources. - Queries are optimized using DataFusion's query optimizer. - Execute user-defined Python code from SQL. - Exchange data with Pandas and other DataFrame libraries that support PyArrow. - Serialize and deserialize query plans in Substrait format. - Experimental support for transpiling SQL queries to DataFrame calls with Polars, Pandas, and cuDF. For tips on tuning parallelism, see [Maximizing CPU Usage](docs/source/user-guide/configuration.rst#maximizing-cpu-usage) in the configuration guide. ## Example Usage The following example demonstrates running a SQL query against a Parquet file using DataFusion, storing the results in a Pandas DataFrame, and then plotting a chart. The Parquet file used in this example can be downloaded from the following page: - https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page ```python from datafusion import SessionContext # Create a DataFusion context ctx = SessionContext() # Register table with context ctx.register_parquet('taxi', 'yellow_tripdata_2021-01.parquet') # Execute SQL df = ctx.sql("select passenger_count, count(*) " "from taxi " "where passenger_count is not null " "group by passenger_count " "order by passenger_count") # convert to Pandas pandas_df = df.to_pandas() # create a chart fig = pandas_df.plot(kind="bar", title="Trip Count by Number of Passengers").get_figure() fig.savefig('chart.png') ``` This produces the following chart: ![Chart](examples/chart.png) ## Registering a DataFrame as a View You can use SessionContext's `register_view` method to convert a DataFrame into a view and register it with the context. ```python from datafusion import SessionContext, col, literal # Create a DataFusion context ctx = SessionContext() # Create sample data data = {"a": [1, 2, 3, 4, 5], "b": [10, 20, 30, 40, 50]} # Create a DataFrame from the dictionary df = ctx.from_pydict(data, "my_table") # Filter the DataFrame (for example, keep rows where a > 2) df_filtered = df.filter(col("a") > literal(2)) # Register the dataframe as a view with the context ctx.register_view("view1", df_filtered) # Now run a SQL query against the registered view df_view = ctx.sql("SELECT * FROM view1") # Collect the results results = df_view.collect() # Convert results to a list of dictionaries for display result_dicts = [batch.to_pydict() for batch in results] print(result_dicts) ``` This will output: ```python [{'a': [3, 4, 5], 'b': [30, 40, 50]}] ``` ## Configuration It is possible to configure runtime (memory and disk settings) and configuration settings when creating a context. ```python runtime = ( RuntimeEnvBuilder() .with_disk_manager_os() .with_fair_spill_pool(10000000) ) config = ( SessionConfig() .with_create_default_catalog_and_schema(True) .with_default_catalog_and_schema("foo", "bar") .with_target_partitions(8) .with_information_schema(True) .with_repartition_joins(False) .with_repartition_aggregations(False) .with_repartition_windows(False) .with_parquet_pruning(False) .set("datafusion.execution.parquet.pushdown_filters", "true") ) ctx = SessionContext(config, runtime) ``` Refer to the [API documentation](https://arrow.apache.org/datafusion-python/#api-reference) for more information. Printing the context will show the current configuration settings. ```python print(ctx) ``` ## Extensions For information about how to extend DataFusion Python, please see the extensions page of the [online documentation](https://datafusion.apache.org/python/). ## More Examples See [examples](examples/README.md) for more information. ### Executing Queries with DataFusion - [Query a Parquet file using SQL](https://github.com/apache/datafusion-python/blob/main/examples/sql-parquet.py) - [Query a Parquet file using the DataFrame API](https://github.com/apache/datafusion-python/blob/main/examples/dataframe-parquet.py) - [Run a SQL query and store the results in a Pandas DataFrame](https://github.com/apache/datafusion-python/blob/main/examples/sql-to-pandas.py) - [Run a SQL query with a Python user-defined function (UDF)](https://github.com/apache/datafusion-python/blob/main/examples/sql-using-python-udf.py) - [Run a SQL query with a Python user-defined aggregation function (UDAF)](https://github.com/apache/datafusion-python/blob/main/examples/sql-using-python-udaf.py) - [Query PyArrow Data](https://github.com/apache/datafusion-python/blob/main/examples/query-pyarrow-data.py) - [Create dataframe](https://github.com/apache/datafusion-python/blob/main/examples/import.py) - [Export dataframe](https://github.com/apache/datafusion-python/blob/main/examples/export.py) ### Running User-Defined Python Code - [Register a Python UDF with DataFusion](https://github.com/apache/datafusion-python/blob/main/examples/python-udf.py) - [Register a Python UDAF with DataFusion](https://github.com/apache/datafusion-python/blob/main/examples/python-udaf.py) ### Substrait Support - [Serialize query plans using Substrait](https://github.com/apache/datafusion-python/blob/main/examples/substrait.py) ## How to install ### uv ```bash uv add datafusion ``` ### Pip ```bash pip install datafusion # or python -m pip install datafusion ``` ### Conda ```bash conda install -c conda-forge datafusion ``` You can verify the installation by running: ```python >>> import datafusion >>> datafusion.__version__ '0.6.0' ``` ## Using DataFusion with AI coding assistants This project ships a [`SKILL.md`](skills/datafusion_python/SKILL.md) that teaches AI coding assistants how to write idiomatic DataFusion Python. It follows the [Agent Skills](https://agentskills.io) open standard. **Preferred:** `npx skills add apache/datafusion-python` — installs the skill in Claude Code, Cursor, Windsurf, Cline, Codex, Copilot, Gemini CLI, and other supported agents. **Manual:** paste this line into your project's `AGENTS.md` / `CLAUDE.md`: ``` For DataFusion Python code, see https://github.com/apache/datafusion-python/blob/main/skills/datafusion_python/SKILL.md ``` ## How to develop This assumes that you have rust and cargo installed. We use the workflow recommended by [pyo3](https://github.com/PyO3/pyo3) and [maturin](https://github.com/PyO3/maturin). The Maturin tools used in this workflow can be installed either via `uv` or `pip`. Both approaches should offer the same experience. It is recommended to use `uv` since it has significant performance improvements over `pip`. Currently for protobuf support either [protobuf](https://protobuf.dev/installation/) or cmake must be installed. Bootstrap (`uv`): By default `uv` will attempt to build the datafusion python package. For our development we prefer to build manually. This means that when creating your virtual environment using `uv sync` you need to pass in the additional `--no-install-package datafusion` and for `uv run` commands the additional parameter `--no-project` ```bash # fetch this repo git clone git@github.com:apache/datafusion-python.git # cd to the repo root cd datafusion-python/ # create the virtual environment uv sync --dev --no-install-package datafusion # activate the environment source .venv/bin/activate ``` Bootstrap (`pip`): ```bash # fetch this repo git clone git@github.com:apache/datafusion-python.git # cd to the repo root cd datafusion-python/ # prepare development environment (used to build wheel / install in development) python3 -m venv .venv # activate the venv source .venv/bin/activate # update pip itself if necessary python -m pip install -U pip # install dependencies python -m pip install -r pyproject.toml ``` The tests rely on test data in git submodules. ```bash git submodule update --init ``` Whenever rust code changes (your changes or via `git pull`): ```bash # make sure you activate the venv using "source venv/bin/activate" first maturin develop --uv python -m pytest ``` Alternatively if you are using `uv` you can do the following without needing to activate the virtual environment: ```bash uv run --no-project maturin develop --uv uv run --no-project pytest ``` To run the FFI tests within the examples folder, after you have built `datafusion-python` with the previous commands: ```bash cd examples/datafusion-ffi-example uv run --no-project maturin develop --uv uv run --no-project pytest python/tests/_test_*py ``` ### Running & Installing pre-commit hooks `datafusion-python` takes advantage of [pre-commit](https://pre-commit.com/) to assist developers with code linting to help reduce the number of commits that ultimately fail in CI due to linter errors. Using the pre-commit hooks is optional for the developer but certainly helpful for keeping PRs clean and concise. Our pre-commit hooks can be installed by running `pre-commit install`, which will install the configurations in your DATAFUSION_PYTHON_ROOT/.github directory and run each time you perform a commit, failing to complete the commit if an offending lint is found allowing you to make changes locally before pushing. The pre-commit hooks can also be run adhoc without installing them by simply running `pre-commit run --all-files`. NOTE: the current `pre-commit` hooks require docker, and cmake. See note on protobuf above. ## Running linters without using pre-commit There are scripts in `ci/scripts` for running Rust and Python linters. ```shell ./ci/scripts/python_lint.sh ./ci/scripts/rust_clippy.sh ./ci/scripts/rust_fmt.sh ./ci/scripts/rust_toml_fmt.sh ``` ## Checking Upstream DataFusion Coverage This project includes an [AI agent skill](.ai/skills/check-upstream/SKILL.md) for auditing which features from the upstream Apache DataFusion Rust library are not yet exposed in these Python bindings. This is useful when adding missing functions, auditing API coverage, or ensuring parity with upstream. The skill accepts an optional area argument: ``` scalar functions aggregate functions window functions dataframe session context ffi types all ``` If no argument is provided, it defaults to checking all areas. The skill will fetch the upstream DataFusion documentation, compare it against the functions and methods exposed in this project, and produce a coverage report listing what is currently exposed and what is missing. The skill definition lives in `.ai/skills/check-upstream/SKILL.md` and follows the [Agent Skills](https://agentskills.io) open standard. It can be used by any AI coding agent that supports skill discovery, or followed manually. ## How to update dependencies To change test dependencies, change the `pyproject.toml` and run ```bash uv sync --dev --no-install-package datafusion ``` apache-datafusion-python-54.0.0/benchmarks/000077500000000000000000000000001521727353300206205ustar00rootroot00000000000000apache-datafusion-python-54.0.0/benchmarks/db-benchmark/000077500000000000000000000000001521727353300231355ustar00rootroot00000000000000apache-datafusion-python-54.0.0/benchmarks/db-benchmark/README.md000066400000000000000000000022271521727353300244170ustar00rootroot00000000000000 # DataFusion Implementation of db-benchmark This directory contains scripts for running [db-benchmark](https://github.com/duckdblabs/db-benchmark) with DataFusion's Python bindings. ## Directions Run the following from root of this project. ```bash docker build -t db-benchmark -f benchmarks/db-benchmark/db-benchmark.dockerfile . docker run --privileged -it db-benchmark ``` apache-datafusion-python-54.0.0/benchmarks/db-benchmark/db-benchmark.dockerfile000066400000000000000000000106371521727353300275120ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. FROM ubuntu:22.04 ARG DEBIAN_FRONTEND=noninteractive ARG TARGETPLATFORM # This section is based on https://github.com/duckdblabs/db-benchmark/blob/master/_utils/repro.sh RUN apt-get -qq update RUN apt-get -qq -y upgrade RUN apt-get -qq install -y apt-utils RUN apt-get -qq install -y lsb-release software-properties-common wget curl vim htop git byobu libcurl4-openssl-dev libssl-dev RUN apt-get -qq install -y libfreetype6-dev RUN apt-get -qq install -y libfribidi-dev RUN apt-get -qq install -y libharfbuzz-dev RUN apt-get -qq install -y git RUN apt-get -qq install -y libxml2-dev RUN apt-get -qq install -y make RUN apt-get -qq install -y libfontconfig1-dev RUN apt-get -qq install -y libicu-dev pandoc zlib1g-dev libgit2-dev libcurl4-openssl-dev libssl-dev libjpeg-dev libpng-dev libtiff-dev # apt-key adv --keyserver keyserver.ubuntu.com --recv-keys E298A3A825C0D65DFD57CBB651716619E084DAB9 RUN add-apt-repository "deb [arch=amd64,i386] https://cloud.r-project.org/bin/linux/ubuntu $(lsb_release -cs)-cran40/" RUN apt-get -qq install -y r-base-dev virtualenv RUN cd /usr/local/lib/R && \ chmod o+w site-library RUN cd / && \ git clone https://github.com/duckdblabs/db-benchmark.git WORKDIR /db-benchmark RUN mkdir -p .R && \ echo 'CFLAGS=-O3 -mtune=native' >> .R/Makevars && \ echo 'CXXFLAGS=-O3 -mtune=native' >> .R/Makevars RUN cd pydatatable && \ virtualenv py-pydatatable --python=/usr/bin/python3.10 RUN cd pandas && \ virtualenv py-pandas --python=/usr/bin/python3.10 RUN cd modin && \ virtualenv py-modin --python=/usr/bin/python3.10 RUN Rscript -e 'install.packages(c("jsonlite","bit64","devtools","rmarkdown"), dependencies=TRUE, repos="https://cloud.r-project.org")' SHELL ["/bin/bash", "-c"] RUN source ./pandas/py-pandas/bin/activate && \ python3 -m pip install --upgrade psutil && \ python3 -m pip install --upgrade pandas && \ deactivate RUN source ./modin/py-modin/bin/activate && \ python3 -m pip install --upgrade modin && \ deactivate RUN source ./pydatatable/py-pydatatable/bin/activate && \ python3 -m pip install --upgrade git+https://github.com/h2oai/datatable && \ deactivate ## install dplyr #RUN Rscript -e 'devtools::install_github(c("tidyverse/readr","tidyverse/dplyr"))' # install data.table RUN Rscript -e 'install.packages("data.table", repos="https://rdatatable.gitlab.io/data.table/")' ## generate data for groupby 0.5GB RUN Rscript _data/groupby-datagen.R 1e7 1e2 0 0 RUN #Rscript _data/groupby-datagen.R 1e8 1e2 0 0 RUN #Rscript _data/groupby-datagen.R 1e9 1e2 0 0 RUN mkdir data && \ mv G1_1e7_1e2_0_0.csv data/ # set only groupby task RUN echo "Changing run.conf and _control/data.csv to run only groupby at 0.5GB" && \ cp run.conf run.conf.original && \ sed -i 's/groupby join groupby2014/groupby/g' run.conf && \ sed -i 's/data.table dplyr pandas pydatatable spark dask clickhouse polars arrow duckdb/data.table dplyr duckdb/g' run.conf && \ sed -i 's/DO_PUBLISH=true/DO_PUBLISH=false/g' run.conf ## set sizes RUN mv _control/data.csv _control/data.csv.original && \ echo "task,data,nrow,k,na,sort,active" > _control/data.csv && \ echo "groupby,G1_1e7_1e2_0_0,1e7,1e2,0,0,1" >> _control/data.csv RUN #./dplyr/setup-dplyr.sh RUN #./datatable/setup-datatable.sh RUN #./duckdb/setup-duckdb.sh # END OF SETUP RUN python3 -m pip install --upgrade pandas RUN python3 -m pip install --upgrade polars psutil RUN python3 -m pip install --upgrade datafusion # Now add our solution RUN rm -rf datafusion-python 2>/dev/null && \ mkdir datafusion-python ADD benchmarks/db-benchmark/*.py datafusion-python/ ADD benchmarks/db-benchmark/run-bench.sh . ENTRYPOINT [ "/db-benchmark/run-bench.sh" ]apache-datafusion-python-54.0.0/benchmarks/db-benchmark/groupby-datafusion.py000066400000000000000000000276571521727353300273520ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import gc import os import timeit from pathlib import Path import datafusion as df import pyarrow as pa from datafusion import ( RuntimeEnvBuilder, SessionConfig, SessionContext, col, ) from datafusion import ( functions as f, ) from pyarrow import csv as pacsv print("# groupby-datafusion.py", flush=True) exec(Path.open("./_helpers/helpers.py").read()) def ans_shape(batches) -> tuple[int, int]: rows, cols = 0, 0 for batch in batches: rows += batch.num_rows if cols == 0: cols = batch.num_columns else: assert cols == batch.num_columns return rows, cols def execute(df) -> list: print(df.execution_plan().display_indent()) return df.collect() ver = df.__version__ git = "" task = "groupby" solution = "datafusion" fun = ".groupby" cache = "TRUE" on_disk = "FALSE" # experimental - support running with both DataFrame and SQL APIs sql = True data_name = os.environ["SRC_DATANAME"] src_grp = "data" / data_name / ".csv" print("loading dataset %s" % src_grp, flush=True) schema = pa.schema( [ ("id4", pa.int32()), ("id5", pa.int32()), ("id6", pa.int32()), ("v1", pa.int32()), ("v2", pa.int32()), ("v3", pa.float64()), ] ) data = pacsv.read_csv( src_grp, convert_options=pacsv.ConvertOptions(auto_dict_encode=True, column_types=schema), ) print("dataset loaded") # create a session context with explicit runtime and config settings runtime = ( RuntimeEnvBuilder() .with_disk_manager_os() .with_fair_spill_pool(64 * 1024 * 1024 * 1024) ) config = ( SessionConfig() .with_repartition_joins(enabled=False) .with_repartition_aggregations(enabled=False) .set("datafusion.execution.coalesce_batches", "false") ) ctx = SessionContext(config, runtime) print(ctx) ctx.register_record_batches("x", [data.to_batches()]) print("registered record batches") # cols = ctx.sql("SHOW columns from x") # ans.show() in_rows = data.num_rows # print(in_rows, flush=True) task_init = timeit.default_timer() question = "sum v1 by id1" # q1 gc.collect() t_start = timeit.default_timer() if sql: df = ctx.sql("SELECT id1, SUM(v1) AS v1 FROM x GROUP BY id1") else: df = ctx.table("x").aggregate([f.col("id1")], [f.sum(f.col("v1")).alias("v1")]) ans = execute(df) shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q1: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "sum v1 by id1:id2" # q2 gc.collect() t_start = timeit.default_timer() if sql: df = ctx.sql("SELECT id1, id2, SUM(v1) AS v1 FROM x GROUP BY id1, id2") else: df = ctx.table("x").aggregate( [f.col("id1"), f.col("id2")], [f.sum(f.col("v1")).alias("v1")] ) ans = execute(df) shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q2: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "sum v1 mean v3 by id3" # q3 gc.collect() t_start = timeit.default_timer() if sql: df = ctx.sql("SELECT id3, SUM(v1) AS v1, AVG(v3) AS v3 FROM x GROUP BY id3") else: df = ctx.table("x").aggregate( [f.col("id3")], [ f.sum(f.col("v1")).alias("v1"), f.avg(f.col("v3")).alias("v3"), ], ) ans = execute(df) shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q3: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = ( df.aggregate([], [f.sum(col("v1")), f.sum(col("v3"))]) .collect()[0] .to_pandas() .to_numpy()[0] ) chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "mean v1:v3 by id4" # q4 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT id4, AVG(v1) AS v1, AVG(v2) AS v2, AVG(v3) AS v3 FROM x GROUP BY id4" ).collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q4: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = ( df.aggregate([], [f.sum(col("v1")), f.sum(col("v2")), f.sum(col("v3"))]) .collect()[0] .to_pandas() .to_numpy()[0] ) chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "sum v1:v3 by id6" # q5 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT id6, SUM(v1) AS v1, SUM(v2) AS v2, SUM(v3) AS v3 FROM x GROUP BY id6" ).collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q5: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = ( df.aggregate([], [f.sum(col("v1")), f.sum(col("v2")), f.sum(col("v3"))]) .collect()[0] .to_pandas() .to_numpy()[0] ) chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "median v3 sd v3 by id4 id5" # q6 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT id4, id5, approx_percentile_cont(v3, .5) AS median_v3, stddev(v3) AS stddev_v3 FROM x GROUP BY id4, id5" ).collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q6: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = ( df.aggregate([], [f.sum(col("median_v3")), f.sum(col("stddev_v3"))]) .collect()[0] .to_pandas() .to_numpy()[0] ) chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "max v1 - min v2 by id3" # q7 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT id3, MAX(v1) - MIN(v2) AS range_v1_v2 FROM x GROUP BY id3" ).collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q7: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("range_v1_v2"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "largest two v3 by id6" # q8 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT id6, v3 from (SELECT id6, v3, row_number() OVER (PARTITION BY id6 ORDER BY v3 DESC) AS row FROM x) t WHERE row <= 2" ).collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q8: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v3"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "regression v1 v2 by id2 id4" # q9 gc.collect() t_start = timeit.default_timer() ans = ctx.sql("SELECT corr(v1, v2) as corr FROM x GROUP BY id2, id4").collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q9: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("corr"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "sum v3 count by id1:id6" # q10 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT id1, id2, id3, id4, id5, id6, SUM(v3) as v3, COUNT(*) AS cnt FROM x GROUP BY id1, id2, id3, id4, id5, id6" ).collect() shape = ans_shape(ans) print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q10: {t}") m = memory_usage() t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = ( df.aggregate([], [f.sum(col("v3")), f.sum(col("cnt"))]) .collect()[0] .to_pandas() .to_numpy()[0] ) chkt = timeit.default_timer() - t_start write_log( task=task, data=data_name, in_rows=in_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() print( "grouping finished, took %0.fs" % (timeit.default_timer() - task_init), flush=True, ) exit(0) apache-datafusion-python-54.0.0/benchmarks/db-benchmark/join-datafusion.py000077500000000000000000000200331521727353300266020ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import gc import os import timeit from pathlib import Path import datafusion as df from datafusion import col from datafusion import functions as f from pyarrow import csv as pacsv print("# join-datafusion.py", flush=True) exec(Path.open("./_helpers/helpers.py").read()) def ans_shape(batches) -> tuple[int, int]: rows, cols = 0, 0 for batch in batches: rows += batch.num_rows if cols == 0: cols = batch.num_columns else: assert cols == batch.num_columns return rows, cols ver = df.__version__ task = "join" git = "" solution = "datafusion" fun = ".join" cache = "TRUE" on_disk = "FALSE" data_name = os.environ["SRC_DATANAME"] src_jn_x = "data" / data_name / ".csv" y_data_name = join_to_tbls(data_name) src_jn_y = [ "data" / y_data_name[0] / ".csv", "data" / y_data_name[1] / ".csv", "data" / y_data_name[2] / ".csv", ] if len(src_jn_y) != 3: error_msg = "Something went wrong in preparing files used for join" raise Exception(error_msg) print( "loading datasets " + data_name + ", " + y_data_name[0] + ", " + y_data_name[1] + ", " + y_data_name[2], flush=True, ) ctx = df.SessionContext() print(ctx) # TODO we should be applying projections to these table reads to create relations # of different sizes x_data = pacsv.read_csv( src_jn_x, convert_options=pacsv.ConvertOptions(auto_dict_encode=True) ) ctx.register_record_batches("x", [x_data.to_batches()]) small_data = pacsv.read_csv( src_jn_y[0], convert_options=pacsv.ConvertOptions(auto_dict_encode=True) ) ctx.register_record_batches("small", [small_data.to_batches()]) medium_data = pacsv.read_csv( src_jn_y[1], convert_options=pacsv.ConvertOptions(auto_dict_encode=True) ) ctx.register_record_batches("medium", [medium_data.to_batches()]) large_data = pacsv.read_csv( src_jn_y[2], convert_options=pacsv.ConvertOptions(auto_dict_encode=True) ) ctx.register_record_batches("large", [large_data.to_batches()]) print(x_data.num_rows, flush=True) print(small_data.num_rows, flush=True) print(medium_data.num_rows, flush=True) print(large_data.num_rows, flush=True) task_init = timeit.default_timer() print("joining...", flush=True) question = "small inner on int" # q1 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT x.id1, x.id2, x.id3, x.id4 as xid4, small.id4 as smallid4, x.id5, x.id6, x.v1, small.v2 FROM x INNER JOIN small ON x.id1 = small.id1" ).collect() # ans = ctx.sql("SELECT * FROM x INNER JOIN small ON x.id1 = small.id1").collect() # print(set([b.schema for b in ans])) shape = ans_shape(ans) # print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q1: {t}") t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start m = memory_usage() write_log( task=task, data=data_name, in_rows=x_data.num_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "medium inner on int" # q2 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT x.id1 as xid1, medium.id1 as mediumid1, x.id2, x.id3, x.id4 as xid4, medium.id4 as mediumid4, x.id5 as xid5, medium.id5 as mediumid5, x.id6, x.v1, medium.v2 FROM x INNER JOIN medium ON x.id2 = medium.id2" ).collect() shape = ans_shape(ans) # print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q2: {t}") t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1")), f.sum(col("v2"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start m = memory_usage() write_log( task=task, data=data_name, in_rows=x_data.num_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "medium outer on int" # q3 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT x.id1 as xid1, medium.id1 as mediumid1, x.id2, x.id3, x.id4 as xid4, medium.id4 as mediumid4, x.id5 as xid5, medium.id5 as mediumid5, x.id6, x.v1, medium.v2 FROM x LEFT JOIN medium ON x.id2 = medium.id2" ).collect() shape = ans_shape(ans) # print(shape, flush=True) t = timeit.default_timer() - t_start print(f"q3: {t}") t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1")), f.sum(col("v2"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start m = memory_usage() write_log( task=task, data=data_name, in_rows=x_data.num_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "medium inner on factor" # q4 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT x.id1 as xid1, medium.id1 as mediumid1, x.id2, x.id3, x.id4 as xid4, medium.id4 as mediumid4, x.id5 as xid5, medium.id5 as mediumid5, x.id6, x.v1, medium.v2 FROM x LEFT JOIN medium ON x.id5 = medium.id5" ).collect() shape = ans_shape(ans) # print(shape) t = timeit.default_timer() - t_start print(f"q4: {t}") t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1")), f.sum(col("v2"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start m = memory_usage() write_log( task=task, data=data_name, in_rows=x_data.num_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() question = "big inner on int" # q5 gc.collect() t_start = timeit.default_timer() ans = ctx.sql( "SELECT x.id1 as xid1, large.id1 as largeid1, x.id2 as xid2, large.id2 as largeid2, x.id3, x.id4 as xid4, large.id4 as largeid4, x.id5 as xid5, large.id5 as largeid5, x.id6 as xid6, large.id6 as largeid6, x.v1, large.v2 FROM x LEFT JOIN large ON x.id3 = large.id3" ).collect() shape = ans_shape(ans) # print(shape) t = timeit.default_timer() - t_start print(f"q5: {t}") t_start = timeit.default_timer() df = ctx.create_dataframe([ans]) chk = df.aggregate([], [f.sum(col("v1")), f.sum(col("v2"))]).collect()[0].column(0)[0] chkt = timeit.default_timer() - t_start m = memory_usage() write_log( task=task, data=data_name, in_rows=x_data.num_rows, question=question, out_rows=shape[0], out_cols=shape[1], solution=solution, version=ver, git=git, fun=fun, run=1, time_sec=t, mem_gb=m, cache=cache, chk=make_chk([chk]), chk_time_sec=chkt, on_disk=on_disk, ) del ans gc.collect() print( "joining finished, took %0.fs" % (timeit.default_timer() - task_init), flush=True, ) exit(0) apache-datafusion-python-54.0.0/benchmarks/db-benchmark/run-bench.sh000077500000000000000000000022301521727353300253520ustar00rootroot00000000000000#!/bin/bash # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. set -e #SRC_DATANAME=G1_1e7_1e2_0_0 python3 /db-benchmark/polars/groupby-polars.py SRC_DATANAME=G1_1e7_1e2_0_0 python3 /db-benchmark/datafusion-python/groupby-datafusion.py # joins need more work still #SRC_DATANAME=G1_1e7_1e2_0_0 python3 /db-benchmark/datafusion-python/join-datafusion.py #SRC_DATANAME=G1_1e7_1e2_0_0 python3 /db-benchmark/polars/join-polars.py cat time.csv apache-datafusion-python-54.0.0/benchmarks/max_cpu_usage.py000066400000000000000000000076701521727353300240240ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Benchmark script showing how to maximize CPU usage. This script demonstrates one example of tuning DataFusion for improved parallelism and CPU utilization. It uses synthetic in-memory data and performs simple aggregation operations to showcase the impact of partitioning configuration. IMPORTANT: This is a simplified example designed to illustrate partitioning concepts. Actual performance in your applications may vary significantly based on many factors: - Type of table providers (Parquet files, CSV, databases, etc.) - I/O operations and storage characteristics (local disk, network, cloud storage) - Query complexity and operation types (joins, window functions, complex expressions) - Data distribution and size characteristics - Memory available and hardware specifications - Network latency for distributed data sources It is strongly recommended that you create similar benchmarks tailored to your specific: - Hardware configuration - Data sources and formats - Typical query patterns and workloads - Performance requirements This will give you more accurate insights into how DataFusion configuration options will affect your particular use case. """ from __future__ import annotations import argparse import multiprocessing import time import pyarrow as pa from datafusion import SessionConfig, SessionContext, col from datafusion import functions as f def main(num_rows: int, partitions: int) -> None: """Run a simple aggregation after repartitioning. This function demonstrates basic partitioning concepts using synthetic data. Real-world performance will depend on your specific data sources, query types, and system configuration. """ # Create some example data (synthetic in-memory data for demonstration) # Note: Real applications typically work with files, databases, or other # data sources that have different I/O and distribution characteristics array = pa.array(range(num_rows)) batch = pa.record_batch([array], names=["a"]) # Configure the session to use a higher target partition count and # enable automatic repartitioning. config = ( SessionConfig() .with_target_partitions(partitions) .with_repartition_joins(enabled=True) .with_repartition_aggregations(enabled=True) .with_repartition_windows(enabled=True) ) ctx = SessionContext(config) # Register the input data and repartition manually to ensure that all # partitions are used. df = ctx.create_dataframe([[batch]]).repartition(partitions) start = time.time() df = df.aggregate([], [f.sum(col("a"))]) df.collect() end = time.time() print( f"Processed {num_rows} rows using {partitions} partitions in {end - start:.3f}s" ) if __name__ == "__main__": parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( "--rows", type=int, default=1_000_000, help="Number of rows in the generated dataset", ) parser.add_argument( "--partitions", type=int, default=multiprocessing.cpu_count(), help="Target number of partitions to use", ) args = parser.parse_args() main(args.rows, args.partitions) apache-datafusion-python-54.0.0/benchmarks/tpch/000077500000000000000000000000001521727353300215565ustar00rootroot00000000000000apache-datafusion-python-54.0.0/benchmarks/tpch/.gitignore000066400000000000000000000000201521727353300235360ustar00rootroot00000000000000data results.csvapache-datafusion-python-54.0.0/benchmarks/tpch/README.md000066400000000000000000000031001521727353300230270ustar00rootroot00000000000000 # DataFusion Python Benchmarks Derived from TPC-H ## Create Release Build From repo root: ```bash maturin develop --release ``` Note that release builds take a really long time, so you may want to temporarily comment out this section of the root Cargo.toml when frequently building. ```toml [profile.release] lto = true codegen-units = 1 ``` ## Generate Data ```bash ./tpch-gen.sh 1 ``` ## Run Benchmarks ```bash python tpch.py ./data ./queries ``` A summary of the benchmark timings will be written to `results.csv`. For example: ```csv setup,1.4 q1,2978.6 q2,679.7 q3,2943.7 q4,2894.9 q5,3592.3 q6,1691.4 q7,3003.9 q8,3818.7 q9,4237.9 q10,2344.7 q11,526.1 q12,2284.6 q13,1009.2 q14,1738.4 q15,1942.1 q16,499.8 q17,5178.9 q18,4127.7 q19,2056.6 q20,2162.5 q21,8046.5 q22,754.9 total,58513.2 ```apache-datafusion-python-54.0.0/benchmarks/tpch/create_tables.sql000066400000000000000000000066751521727353300251120ustar00rootroot00000000000000-- Schema derived from TPC-H schema under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. CREATE EXTERNAL TABLE customer ( c_custkey INT NOT NULL, c_name VARCHAR NOT NULL, c_address VARCHAR NOT NULL, c_nationkey INT NOT NULL, c_phone VARCHAR NOT NULL, c_acctbal DECIMAL(15, 2) NOT NULL, c_mktsegment VARCHAR NOT NULL, c_comment VARCHAR NOT NULL, c_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/customer.csv'; CREATE EXTERNAL TABLE lineitem ( l_orderkey INT NOT NULL, l_partkey INT NOT NULL, l_suppkey INT NOT NULL, l_linenumber INT NOT NULL, l_quantity DECIMAL(15, 2) NOT NULL, l_extendedprice DECIMAL(15, 2) NOT NULL, l_discount DECIMAL(15, 2) NOT NULL, l_tax DECIMAL(15, 2) NOT NULL, l_returnflag VARCHAR NOT NULL, l_linestatus VARCHAR NOT NULL, l_shipdate DATE NOT NULL, l_commitdate DATE NOT NULL, l_receiptdate DATE NOT NULL, l_shipinstruct VARCHAR NOT NULL, l_shipmode VARCHAR NOT NULL, l_comment VARCHAR NOT NULL, l_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/lineitem.csv'; CREATE EXTERNAL TABLE nation ( n_nationkey INT NOT NULL, n_name VARCHAR NOT NULL, n_regionkey INT NOT NULL, n_comment VARCHAR NOT NULL, n_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/nation.csv'; CREATE EXTERNAL TABLE orders ( o_orderkey INT NOT NULL, o_custkey INT NOT NULL, o_orderstatus VARCHAR NOT NULL, o_totalprice DECIMAL(15, 2) NOT NULL, o_orderdate DATE NOT NULL, o_orderpriority VARCHAR NOT NULL, o_clerk VARCHAR NOT NULL, o_shippriority INT NULL, o_comment VARCHAR NOT NULL, o_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/orders.csv'; CREATE EXTERNAL TABLE part ( p_partkey INT NOT NULL, p_name VARCHAR NOT NULL, p_mfgr VARCHAR NOT NULL, p_brand VARCHAR NOT NULL, p_type VARCHAR NOT NULL, p_size INT NULL, p_container VARCHAR NOT NULL, p_retailprice DECIMAL(15, 2) NOT NULL, p_comment VARCHAR NOT NULL, p_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/part.csv'; CREATE EXTERNAL TABLE partsupp ( ps_partkey INT NOT NULL, ps_suppkey INT NOT NULL, ps_availqty INT NOT NULL, ps_supplycost DECIMAL(15, 2) NOT NULL, ps_comment VARCHAR NOT NULL, ps_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/partsupp.csv'; CREATE EXTERNAL TABLE region ( r_regionkey INT NOT NULL, r_name VARCHAR NOT NULL, r_comment VARCHAR NOT NULL, r_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/region.csv'; CREATE EXTERNAL TABLE supplier ( s_suppkey INT NOT NULL, s_name VARCHAR NOT NULL, s_address VARCHAR NOT NULL, s_nationkey INT NOT NULL, s_phone VARCHAR NOT NULL, s_acctbal DECIMAL(15, 2) NOT NULL, s_comment VARCHAR NOT NULL, s_extra VARCHAR NOT NULL, ) STORED AS CSV OPTIONS ( format.delimiter '|', format.has_header true ) LOCATION '$PATH/supplier.csv';apache-datafusion-python-54.0.0/benchmarks/tpch/queries/000077500000000000000000000000001521727353300232335ustar00rootroot00000000000000apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q1.sql000066400000000000000000000012561521727353300243010ustar00rootroot00000000000000-- Benchmark Query 1 derived from TPC-H query 1 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select l_returnflag, l_linestatus, sum(l_quantity) as sum_qty, sum(l_extendedprice) as sum_base_price, sum(l_extendedprice * (1 - l_discount)) as sum_disc_price, sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) as sum_charge, avg(l_quantity) as avg_qty, avg(l_extendedprice) as avg_price, avg(l_discount) as avg_disc, count(*) as count_order from lineitem where l_shipdate <= date '1998-12-01' - interval '68 days' group by l_returnflag, l_linestatus order by l_returnflag, l_linestatus; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q10.sql000066400000000000000000000012541521727353300243570ustar00rootroot00000000000000-- Benchmark Query 10 derived from TPC-H query 10 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select c_custkey, c_name, sum(l_extendedprice * (1 - l_discount)) as revenue, c_acctbal, n_name, c_address, c_phone, c_comment from customer, orders, lineitem, nation where c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate >= date '1993-07-01' and o_orderdate < date '1993-07-01' + interval '3' month and l_returnflag = 'R' and c_nationkey = n_nationkey group by c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment order by revenue desc limit 20; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q11.sql000066400000000000000000000012251521727353300243560ustar00rootroot00000000000000-- Benchmark Query 11 derived from TPC-H query 11 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select ps_partkey, sum(ps_supplycost * ps_availqty) as value from partsupp, supplier, nation where ps_suppkey = s_suppkey and s_nationkey = n_nationkey and n_name = 'ALGERIA' group by ps_partkey having sum(ps_supplycost * ps_availqty) > ( select sum(ps_supplycost * ps_availqty) * 0.0001000000 from partsupp, supplier, nation where ps_suppkey = s_suppkey and s_nationkey = n_nationkey and n_name = 'ALGERIA' ) order by value desc; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q12.sql000066400000000000000000000013511521727353300243570ustar00rootroot00000000000000-- Benchmark Query 12 derived from TPC-H query 12 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select l_shipmode, sum(case when o_orderpriority = '1-URGENT' or o_orderpriority = '2-HIGH' then 1 else 0 end) as high_line_count, sum(case when o_orderpriority <> '1-URGENT' and o_orderpriority <> '2-HIGH' then 1 else 0 end) as low_line_count from orders, lineitem where o_orderkey = l_orderkey and l_shipmode in ('FOB', 'SHIP') and l_commitdate < l_receiptdate and l_shipdate < l_commitdate and l_receiptdate >= date '1995-01-01' and l_receiptdate < date '1995-01-01' + interval '1' year group by l_shipmode order by l_shipmode; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q13.sql000066400000000000000000000007731521727353300243670ustar00rootroot00000000000000-- Benchmark Query 13 derived from TPC-H query 13 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select c_count, count(*) as custdist from ( select c_custkey, count(o_orderkey) from customer left outer join orders on c_custkey = o_custkey and o_comment not like '%express%requests%' group by c_custkey ) as c_orders (c_custkey, c_count) group by c_count order by custdist desc, c_count desc; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q14.sql000066400000000000000000000007601521727353300243640ustar00rootroot00000000000000-- Benchmark Query 14 derived from TPC-H query 14 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select 100.00 * sum(case when p_type like 'PROMO%' then l_extendedprice * (1 - l_discount) else 0 end) / sum(l_extendedprice * (1 - l_discount)) as promo_revenue from lineitem, part where l_partkey = p_partkey and l_shipdate >= date '1995-02-01' and l_shipdate < date '1995-02-01' + interval '1' month; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q15.sql000066400000000000000000000012411521727353300243600ustar00rootroot00000000000000-- Benchmark Query 15 derived from TPC-H query 15 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. create view revenue0 (supplier_no, total_revenue) as select l_suppkey, sum(l_extendedprice * (1 - l_discount)) from lineitem where l_shipdate >= date '1996-08-01' and l_shipdate < date '1996-08-01' + interval '3' month group by l_suppkey; select s_suppkey, s_name, s_address, s_phone, total_revenue from supplier, revenue0 where s_suppkey = supplier_no and total_revenue = ( select max(total_revenue) from revenue0 ) order by s_suppkey; drop view revenue0; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q16.sql000066400000000000000000000011701521727353300243620ustar00rootroot00000000000000-- Benchmark Query 16 derived from TPC-H query 16 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select p_brand, p_type, p_size, count(distinct ps_suppkey) as supplier_cnt from partsupp, part where p_partkey = ps_partkey and p_brand <> 'Brand#14' and p_type not like 'SMALL PLATED%' and p_size in (14, 6, 5, 31, 49, 15, 41, 47) and ps_suppkey not in ( select s_suppkey from supplier where s_comment like '%Customer%Complaints%' ) group by p_brand, p_type, p_size order by supplier_cnt desc, p_brand, p_type, p_size; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q17.sql000066400000000000000000000006731521727353300243720ustar00rootroot00000000000000-- Benchmark Query 17 derived from TPC-H query 17 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select sum(l_extendedprice) / 7.0 as avg_yearly from lineitem, part where p_partkey = l_partkey and p_brand = 'Brand#42' and p_container = 'LG BAG' and l_quantity < ( select 0.2 * avg(l_quantity) from lineitem where l_partkey = p_partkey ); apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q18.sql000066400000000000000000000011371521727353300243670ustar00rootroot00000000000000-- Benchmark Query 18 derived from TPC-H query 18 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where o_orderkey in ( select l_orderkey from lineitem group by l_orderkey having sum(l_quantity) > 313 ) and c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 100; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q19.sql000066400000000000000000000021401521727353300243630ustar00rootroot00000000000000-- Benchmark Query 19 derived from TPC-H query 19 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select sum(l_extendedprice* (1 - l_discount)) as revenue from lineitem, part where ( p_partkey = l_partkey and p_brand = 'Brand#21' and p_container in ('SM CASE', 'SM BOX', 'SM PACK', 'SM PKG') and l_quantity >= 8 and l_quantity <= 8 + 10 and p_size between 1 and 5 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ) or ( p_partkey = l_partkey and p_brand = 'Brand#13' and p_container in ('MED BAG', 'MED BOX', 'MED PKG', 'MED PACK') and l_quantity >= 20 and l_quantity <= 20 + 10 and p_size between 1 and 10 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ) or ( p_partkey = l_partkey and p_brand = 'Brand#52' and p_container in ('LG CASE', 'LG BOX', 'LG PACK', 'LG PKG') and l_quantity >= 30 and l_quantity <= 30 + 10 and p_size between 1 and 15 and l_shipmode in ('AIR', 'AIR REG') and l_shipinstruct = 'DELIVER IN PERSON' ); apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q2.sql000066400000000000000000000014771521727353300243070ustar00rootroot00000000000000-- Benchmark Query 2 derived from TPC-H query 2 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select s_acctbal, s_name, n_name, p_partkey, p_mfgr, s_address, s_phone, s_comment from part, supplier, partsupp, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and p_size = 48 and p_type like '%TIN' and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'ASIA' and ps_supplycost = ( select min(ps_supplycost) from partsupp, supplier, nation, region where p_partkey = ps_partkey and s_suppkey = ps_suppkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'ASIA' ) order by s_acctbal desc, n_name, s_name, p_partkey limit 100; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q20.sql000066400000000000000000000013501521727353300243550ustar00rootroot00000000000000-- Benchmark Query 20 derived from TPC-H query 20 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select s_name, s_address from supplier, nation where s_suppkey in ( select ps_suppkey from partsupp where ps_partkey in ( select p_partkey from part where p_name like 'blanched%' ) and ps_availqty > ( select 0.5 * sum(l_quantity) from lineitem where l_partkey = ps_partkey and l_suppkey = ps_suppkey and l_shipdate >= date '1993-01-01' and l_shipdate < date '1993-01-01' + interval '1' year ) ) and s_nationkey = n_nationkey and n_name = 'KENYA' order by s_name; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q21.sql000066400000000000000000000014571521727353300243660ustar00rootroot00000000000000-- Benchmark Query 21 derived from TPC-H query 21 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select s_name, count(*) as numwait from supplier, lineitem l1, orders, nation where s_suppkey = l1.l_suppkey and o_orderkey = l1.l_orderkey and o_orderstatus = 'F' and l1.l_receiptdate > l1.l_commitdate and exists ( select * from lineitem l2 where l2.l_orderkey = l1.l_orderkey and l2.l_suppkey <> l1.l_suppkey ) and not exists ( select * from lineitem l3 where l3.l_orderkey = l1.l_orderkey and l3.l_suppkey <> l1.l_suppkey and l3.l_receiptdate > l3.l_commitdate ) and s_nationkey = n_nationkey and n_name = 'ARGENTINA' group by s_name order by numwait desc, s_name limit 100; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q22.sql000066400000000000000000000014431521727353300243620ustar00rootroot00000000000000-- Benchmark Query 22 derived from TPC-H query 22 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select cntrycode, count(*) as numcust, sum(c_acctbal) as totacctbal from ( select substring(c_phone from 1 for 2) as cntrycode, c_acctbal from customer where substring(c_phone from 1 for 2) in ('24', '34', '16', '30', '33', '14', '13') and c_acctbal > ( select avg(c_acctbal) from customer where c_acctbal > 0.00 and substring(c_phone from 1 for 2) in ('24', '34', '16', '30', '33', '14', '13') ) and not exists ( select * from orders where o_custkey = c_custkey ) ) as custsale group by cntrycode order by cntrycode; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q3.sql000066400000000000000000000011001521727353300242670ustar00rootroot00000000000000-- Benchmark Query 3 derived from TPC-H query 3 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select l_orderkey, sum(l_extendedprice * (1 - l_discount)) as revenue, o_orderdate, o_shippriority from customer, orders, lineitem where c_mktsegment = 'BUILDING' and c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate < date '1995-03-15' and l_shipdate > date '1995-03-15' group by l_orderkey, o_orderdate, o_shippriority order by revenue desc, o_orderdate limit 10; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q4.sql000066400000000000000000000010001521727353300242670ustar00rootroot00000000000000-- Benchmark Query 4 derived from TPC-H query 4 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select o_orderpriority, count(*) as order_count from orders where o_orderdate >= date '1995-04-01' and o_orderdate < date '1995-04-01' + interval '3' month and exists ( select * from lineitem where l_orderkey = o_orderkey and l_commitdate < l_receiptdate ) group by o_orderpriority order by o_orderpriority; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q5.sql000066400000000000000000000012061521727353300243000ustar00rootroot00000000000000-- Benchmark Query 5 derived from TPC-H query 5 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select n_name, sum(l_extendedprice * (1 - l_discount)) as revenue from customer, orders, lineitem, supplier, nation, region where c_custkey = o_custkey and l_orderkey = o_orderkey and l_suppkey = s_suppkey and c_nationkey = s_nationkey and s_nationkey = n_nationkey and n_regionkey = r_regionkey and r_name = 'AFRICA' and o_orderdate >= date '1994-01-01' and o_orderdate < date '1994-01-01' + interval '1' year group by n_name order by revenue desc; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q6.sql000066400000000000000000000006361521727353300243070ustar00rootroot00000000000000-- Benchmark Query 6 derived from TPC-H query 6 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select sum(l_extendedprice * l_discount) as revenue from lineitem where l_shipdate >= date '1994-01-01' and l_shipdate < date '1994-01-01' + interval '1' year and l_discount between 0.04 - 0.01 and 0.04 + 0.01 and l_quantity < 24; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q7.sql000066400000000000000000000016701521727353300243070ustar00rootroot00000000000000-- Benchmark Query 7 derived from TPC-H query 7 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select supp_nation, cust_nation, l_year, sum(volume) as revenue from ( select n1.n_name as supp_nation, n2.n_name as cust_nation, extract(year from l_shipdate) as l_year, l_extendedprice * (1 - l_discount) as volume from supplier, lineitem, orders, customer, nation n1, nation n2 where s_suppkey = l_suppkey and o_orderkey = l_orderkey and c_custkey = o_custkey and s_nationkey = n1.n_nationkey and c_nationkey = n2.n_nationkey and ( (n1.n_name = 'GERMANY' and n2.n_name = 'IRAQ') or (n1.n_name = 'IRAQ' and n2.n_name = 'GERMANY') ) and l_shipdate between date '1995-01-01' and date '1996-12-31' ) as shipping group by supp_nation, cust_nation, l_year order by supp_nation, cust_nation, l_year; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q8.sql000066400000000000000000000016471521727353300243140ustar00rootroot00000000000000-- Benchmark Query 8 derived from TPC-H query 8 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select o_year, sum(case when nation = 'IRAQ' then volume else 0 end) / sum(volume) as mkt_share from ( select extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) as volume, n2.n_name as nation from part, supplier, lineitem, orders, customer, nation n1, nation n2, region where p_partkey = l_partkey and s_suppkey = l_suppkey and l_orderkey = o_orderkey and o_custkey = c_custkey and c_nationkey = n1.n_nationkey and n1.n_regionkey = r_regionkey and r_name = 'MIDDLE EAST' and s_nationkey = n2.n_nationkey and o_orderdate between date '1995-01-01' and date '1996-12-31' and p_type = 'LARGE PLATED STEEL' ) as all_nations group by o_year order by o_year; apache-datafusion-python-54.0.0/benchmarks/tpch/queries/q9.sql000066400000000000000000000013621521727353300243070ustar00rootroot00000000000000-- Benchmark Query 9 derived from TPC-H query 9 under the terms of the TPC Fair Use Policy. -- TPC-H queries are Copyright 1993-2022 Transaction Processing Performance Council. select nation, o_year, sum(amount) as sum_profit from ( select n_name as nation, extract(year from o_orderdate) as o_year, l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount from part, supplier, lineitem, partsupp, orders, nation where s_suppkey = l_suppkey and ps_suppkey = l_suppkey and ps_partkey = l_partkey and p_partkey = l_partkey and o_orderkey = l_orderkey and s_nationkey = n_nationkey and p_name like '%moccasin%' ) as profit group by nation, o_year order by nation, o_year desc; apache-datafusion-python-54.0.0/benchmarks/tpch/tpch-gen.sh000077500000000000000000000041041521727353300236210ustar00rootroot00000000000000#!/bin/bash # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. mkdir -p data/answers 2>/dev/null set -e # If RUN_IN_CI is set, then do not produce verbose output or use an interactive terminal if [[ -z "${RUN_IN_CI}" ]]; then TERMINAL_FLAG="-it" VERBOSE_OUTPUT="-vf" else TERMINAL_FLAG="" VERBOSE_OUTPUT="-f" fi #pushd .. #. ./dev/build-set-env.sh #popd # Generate data into the ./data directory if it does not already exist FILE=./data/supplier.tbl if test -f "$FILE"; then echo "$FILE exists." else docker run -v `pwd`/data:/data $TERMINAL_FLAG --rm ghcr.io/scalytics/tpch-docker:main $VERBOSE_OUTPUT -s $1 # workaround for https://github.com/apache/arrow-datafusion/issues/6147 mv data/customer.tbl data/customer.csv mv data/lineitem.tbl data/lineitem.csv mv data/nation.tbl data/nation.csv mv data/orders.tbl data/orders.csv mv data/part.tbl data/part.csv mv data/partsupp.tbl data/partsupp.csv mv data/region.tbl data/region.csv mv data/supplier.tbl data/supplier.csv ls -l data fi # Copy expected answers (at SF=1) into the ./data/answers directory if it does not already exist FILE=./data/answers/q1.out if test -f "$FILE"; then echo "$FILE exists." else docker run -v `pwd`/data:/data $TERMINAL_FLAG --entrypoint /bin/bash --rm ghcr.io/scalytics/tpch-docker:main -c "cp /opt/tpch/2.18.0_rc2/dbgen/answers/* /data/answers/" fi apache-datafusion-python-54.0.0/benchmarks/tpch/tpch.py000066400000000000000000000067101521727353300230720ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import argparse import time from pathlib import Path from datafusion import SessionContext def bench(data_path, query_path) -> None: with Path("results.csv").open("w") as results: # register tables start = time.time() total_time_millis = 0 # create context # runtime = ( # RuntimeEnvBuilder() # .with_disk_manager_os() # .with_fair_spill_pool(10000000) # ) # config = ( # SessionConfig() # .with_create_default_catalog_and_schema(True) # .with_default_catalog_and_schema("datafusion", "tpch") # .with_information_schema(True) # ) # ctx = SessionContext(config, runtime) ctx = SessionContext() print("Configuration:\n", ctx) # register tables with Path("create_tables.sql").open() as f: sql = "" for line in f.readlines(): if line.startswith("--"): continue sql = sql + line if sql.strip().endswith(";"): sql = sql.strip().replace("$PATH", data_path) ctx.sql(sql) sql = "" end = time.time() time_millis = (end - start) * 1000 total_time_millis += time_millis print(f"setup,{round(time_millis, 1)}") results.write(f"setup,{round(time_millis, 1)}\n") results.flush() # run queries for query in range(1, 23): with Path(f"{query_path}/q{query}.sql").open() as f: text = f.read() tmp = text.split(";") queries = [s.strip() for s in tmp if len(s.strip()) > 0] try: start = time.time() for sql in queries: print(sql) df = ctx.sql(sql) # result_set = df.collect() df.show() end = time.time() time_millis = (end - start) * 1000 total_time_millis += time_millis print(f"q{query},{round(time_millis, 1)}") results.write(f"q{query},{round(time_millis, 1)}\n") results.flush() except Exception as e: print("query", query, "failed", e) print(f"total,{round(total_time_millis, 1)}") results.write(f"total,{round(total_time_millis, 1)}\n") if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("data_path") parser.add_argument("query_path") args = parser.parse_args() bench(args.data_path, args.query_path) apache-datafusion-python-54.0.0/ci/000077500000000000000000000000001521727353300170765ustar00rootroot00000000000000apache-datafusion-python-54.0.0/ci/scripts/000077500000000000000000000000001521727353300205655ustar00rootroot00000000000000apache-datafusion-python-54.0.0/ci/scripts/python_lint.sh000077500000000000000000000015341521727353300234760ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. set -ex ruff format datafusion ruff check datafusionapache-datafusion-python-54.0.0/ci/scripts/rust_clippy.sh000077500000000000000000000015711521727353300235050ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. set -ex cargo clippy --all-targets --workspace --features default -- -D warnings apache-datafusion-python-54.0.0/ci/scripts/rust_fmt.sh000077500000000000000000000015241521727353300227710ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. set -ex cargo +nightly fmt --all -- --check apache-datafusion-python-54.0.0/ci/scripts/rust_toml_fmt.sh000077500000000000000000000015631521727353300240270ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. set -ex find . -mindepth 2 -name 'Cargo.toml' -exec cargo tomlfmt -p {} \; apache-datafusion-python-54.0.0/conftest.py000066400000000000000000000024761521727353300207130ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Pytest configuration for doctest namespace injection.""" import datafusion as dfn import numpy as np import pyarrow as pa import pytest from datafusion import col, lit from datafusion import functions as F @pytest.fixture(autouse=True) def _doctest_namespace(doctest_namespace: dict) -> None: """Add common imports to the doctest namespace.""" doctest_namespace["dfn"] = dfn doctest_namespace["np"] = np doctest_namespace["pa"] = pa doctest_namespace["col"] = col doctest_namespace["lit"] = lit doctest_namespace["F"] = F apache-datafusion-python-54.0.0/crates/000077500000000000000000000000001521727353300177645ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/000077500000000000000000000000001521727353300207145ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/Cargo.toml000066400000000000000000000056471521727353300226600ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. [package] name = "datafusion-python" version.workspace = true edition.workspace = true rust-version.workspace = true license.workspace = true description.workspace = true homepage.workspace = true repository.workspace = true include = [ "src", "../LICENSE.txt", "build.rs", "../pyproject.toml", "Cargo.toml", "../Cargo.lock", ] [dependencies] tokio = { workspace = true, features = [ "macros", "rt", "rt-multi-thread", "sync", ] } pyo3 = { workspace = true, features = [ "extension-module", "generate-import-lib", ] } pyo3-async-runtimes = { workspace = true, features = ["tokio-runtime"] } pyo3-log = { workspace = true } chrono = { workspace = true } arrow = { workspace = true, features = ["pyarrow"] } arrow-select = { workspace = true } datafusion = { workspace = true, features = ["avro", "unicode_expressions"] } datafusion-substrait = { workspace = true, optional = true } datafusion-proto = { workspace = true } datafusion-ffi = { workspace = true } datafusion-spark = { workspace = true } prost = { workspace = true } # keep in line with `datafusion-substrait` serde_json = { workspace = true } uuid = { workspace = true, features = ["v4"] } mimalloc = { workspace = true, optional = true, features = [ "local_dynamic_tls", # Pin to mimalloc v2 until apache/datafusion-python#1607 resolves. "v2", ] } async-trait = { workspace = true } futures = { workspace = true } cstr = { workspace = true } object_store = { workspace = true, features = ["aws", "gcp", "azure", "http"] } url = { workspace = true } log = { workspace = true } parking_lot = { workspace = true } datafusion-python-util = { workspace = true } [build-dependencies] prost-types = { workspace = true } pyo3-build-config = { workspace = true } [features] default = ["mimalloc", "abi3"] # Stable ABI build — single wheel covers Python 3.10..3.14 (GIL builds only). # Mutually exclusive with free-threaded interpreters (cp313t / cp314t); the # free-threaded wheel build must pass --no-default-features. abi3 = ["pyo3/abi3", "pyo3/abi3-py310"] protoc = ["datafusion-substrait/protoc"] substrait = ["dep:datafusion-substrait"] [lib] name = "datafusion_python" crate-type = ["cdylib", "rlib"] apache-datafusion-python-54.0.0/crates/core/build.rs000066400000000000000000000015511521727353300223630ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. fn main() { pyo3_build_config::add_extension_module_link_args(); } apache-datafusion-python-54.0.0/crates/core/src/000077500000000000000000000000001521727353300215035ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/src/analyzer.rs000066400000000000000000000042251521727353300237010ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Analyzer rules layered on top of DataFusion's defaults. use datafusion::common::Result; use datafusion::common::config::ConfigOptions; use datafusion::logical_expr::LogicalPlan; use datafusion::optimizer::AnalyzerRule; /// Resolve [`LambdaVariable`] references into bound lambda parameters. /// /// DataFusion's SQL planner resolves lambda variables inline as it plans a /// higher-order function call, so SQL-built plans never carry unresolved /// variables. Plans assembled programmatically through the Python expression /// builder (e.g. `array_transform(col("xs"), lambda_(["v"], lambda_var("v")))`) /// do carry them, and nothing in the default analyzer resolves them. This rule /// runs [`LogicalPlan::resolve_lambda_variables`] so both construction paths /// reach the optimizer with bound lambdas. /// /// [`LambdaVariable`]: datafusion::logical_expr::expr::LambdaVariable #[derive(Debug)] pub struct ResolveLambdaVariables {} impl ResolveLambdaVariables { pub fn new() -> Self { Self {} } } impl Default for ResolveLambdaVariables { fn default() -> Self { Self::new() } } impl AnalyzerRule for ResolveLambdaVariables { fn analyze(&self, plan: LogicalPlan, _config: &ConfigOptions) -> Result { plan.resolve_lambda_variables().map(|t| t.data) } fn name(&self) -> &str { "resolve_lambda_variables" } } apache-datafusion-python-54.0.0/crates/core/src/array.rs000066400000000000000000000057171521727353300232010ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ptr::NonNull; use std::sync::Arc; use arrow::array::{Array, ArrayRef}; use arrow::datatypes::{Field, FieldRef}; use arrow::ffi::{FFI_ArrowArray, FFI_ArrowSchema}; use arrow::pyarrow::ToPyArrow; use pyo3::prelude::{PyAnyMethods, PyCapsuleMethods}; use pyo3::types::PyCapsule; use pyo3::{Bound, PyAny, PyResult, Python, pyclass, pymethods}; use crate::errors::PyDataFusionResult; /// A Python object which implements the Arrow PyCapsule for importing /// into other libraries. #[pyclass( from_py_object, name = "ArrowArrayExportable", module = "datafusion", frozen )] #[derive(Clone)] pub struct PyArrowArrayExportable { array: ArrayRef, field: FieldRef, } #[pymethods] impl PyArrowArrayExportable { #[pyo3(signature = (requested_schema=None))] fn __arrow_c_array__<'py>( &'py self, py: Python<'py>, requested_schema: Option>, ) -> PyDataFusionResult<(Bound<'py, PyCapsule>, Bound<'py, PyCapsule>)> { let field = if let Some(schema_capsule) = requested_schema { let data: NonNull = schema_capsule .pointer_checked(Some(c"arrow_schema"))? .cast(); let schema_ptr = unsafe { data.as_ref() }; let desired_field = Field::try_from(schema_ptr)?; Arc::new(desired_field) } else { Arc::clone(&self.field) }; let ffi_schema = FFI_ArrowSchema::try_from(&field)?; let schema_capsule = PyCapsule::new(py, ffi_schema, Some(cr"arrow_schema".into()))?; let ffi_array = FFI_ArrowArray::new(&self.array.to_data()); let array_capsule = PyCapsule::new(py, ffi_array, Some(cr"arrow_array".into()))?; Ok((schema_capsule, array_capsule)) } } impl ToPyArrow for PyArrowArrayExportable { fn to_pyarrow<'py>(&self, py: Python<'py>) -> PyResult> { let module = py.import("pyarrow")?; let method = module.getattr("array")?; let array = method.call((self.clone(),), None)?; Ok(array) } } impl PyArrowArrayExportable { pub fn new(array: ArrayRef, field: FieldRef) -> Self { Self { array, field } } } apache-datafusion-python-54.0.0/crates/core/src/catalog.rs000066400000000000000000000537751521727353300235040ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashSet; use std::ptr::NonNull; use std::sync::Arc; use async_trait::async_trait; use datafusion::catalog::{ CatalogProvider, CatalogProviderList, MemoryCatalogProvider, MemoryCatalogProviderList, MemorySchemaProvider, SchemaProvider, }; use datafusion::common::DataFusionError; use datafusion::datasource::TableProvider; use datafusion_ffi::catalog_provider::FFI_CatalogProvider; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; use datafusion_ffi::schema_provider::FFI_SchemaProvider; use datafusion_python_util::{ create_logical_extension_capsule, ffi_logical_codec_from_pycapsule, wait_for_future, }; use pyo3::IntoPyObjectExt; use pyo3::exceptions::PyKeyError; use pyo3::prelude::*; use pyo3::types::PyCapsule; use crate::context::PySessionContext; use crate::dataset::Dataset; use crate::errors::{PyDataFusionError, PyDataFusionResult, py_datafusion_err, to_datafusion_err}; use crate::table::PyTable; #[pyclass( from_py_object, frozen, name = "RawCatalogList", module = "datafusion.catalog", subclass )] #[derive(Clone)] pub struct PyCatalogList { pub catalog_list: Arc, codec: Arc, } #[pyclass( from_py_object, frozen, name = "RawCatalog", module = "datafusion.catalog", subclass )] #[derive(Clone)] pub struct PyCatalog { pub catalog: Arc, codec: Arc, } #[pyclass( from_py_object, frozen, name = "RawSchema", module = "datafusion.catalog", subclass )] #[derive(Clone)] pub struct PySchema { pub schema: Arc, codec: Arc, } impl PyCatalog { pub(crate) fn new_from_parts( catalog: Arc, codec: Arc, ) -> Self { Self { catalog, codec } } } impl PySchema { pub(crate) fn new_from_parts( schema: Arc, codec: Arc, ) -> Self { Self { schema, codec } } } #[pymethods] impl PyCatalogList { #[new] pub fn new( py: Python, catalog_list: Py, session: Option>, ) -> PyResult { let codec = extract_logical_extension_codec(py, session)?; let catalog_list = Arc::new(RustWrappedPyCatalogProviderList::new( catalog_list, codec.clone(), )) as Arc; Ok(Self { catalog_list, codec, }) } #[staticmethod] pub fn memory_catalog_list(py: Python, session: Option>) -> PyResult { let codec = extract_logical_extension_codec(py, session)?; let catalog_list = Arc::new(MemoryCatalogProviderList::default()) as Arc; Ok(Self { catalog_list, codec, }) } pub fn catalog_names(&self) -> HashSet { self.catalog_list.catalog_names().into_iter().collect() } #[pyo3(signature = (name="public"))] pub fn catalog(&self, name: &str) -> PyResult> { let catalog = self .catalog_list .catalog(name) .ok_or(PyKeyError::new_err(format!( "Schema with name {name} doesn't exist." )))?; Python::attach( |py| match catalog.downcast_ref::() { Some(wrapped_catalog) => Ok(wrapped_catalog.catalog_provider.clone_ref(py)), None => PyCatalog::new_from_parts(catalog, self.codec.clone()).into_py_any(py), }, ) } pub fn register_catalog(&self, name: &str, catalog_provider: Bound<'_, PyAny>) -> PyResult<()> { let provider = extract_catalog_provider_from_pyobj(catalog_provider, self.codec.as_ref())?; let _ = self .catalog_list .register_catalog(name.to_owned(), provider); Ok(()) } pub fn __repr__(&self) -> PyResult { let mut names: Vec = self.catalog_names().into_iter().collect(); names.sort(); Ok(format!("CatalogList(catalog_names=[{}])", names.join(", "))) } } #[pymethods] impl PyCatalog { #[new] pub fn new(py: Python, catalog: Py, session: Option>) -> PyResult { let codec = extract_logical_extension_codec(py, session)?; let catalog = Arc::new(RustWrappedPyCatalogProvider::new(catalog, codec.clone())) as Arc; Ok(Self { catalog, codec }) } #[staticmethod] pub fn memory_catalog(py: Python, session: Option>) -> PyResult { let codec = extract_logical_extension_codec(py, session)?; let catalog = Arc::new(MemoryCatalogProvider::default()) as Arc; Ok(Self { catalog, codec }) } pub fn schema_names(&self) -> HashSet { self.catalog.schema_names().into_iter().collect() } #[pyo3(signature = (name="public"))] pub fn schema(&self, name: &str) -> PyResult> { let schema = self .catalog .schema(name) .ok_or(PyKeyError::new_err(format!( "Schema with name {name} doesn't exist." )))?; Python::attach( |py| match schema.downcast_ref::() { Some(wrapped_schema) => Ok(wrapped_schema.schema_provider.clone_ref(py)), None => PySchema::new_from_parts(schema, self.codec.clone()).into_py_any(py), }, ) } pub fn register_schema(&self, name: &str, schema_provider: Bound<'_, PyAny>) -> PyResult<()> { let provider = extract_schema_provider_from_pyobj(schema_provider, self.codec.as_ref())?; let _ = self .catalog .register_schema(name, provider) .map_err(py_datafusion_err)?; Ok(()) } pub fn deregister_schema(&self, name: &str, cascade: bool) -> PyResult<()> { let _ = self .catalog .deregister_schema(name, cascade) .map_err(py_datafusion_err)?; Ok(()) } pub fn __repr__(&self) -> PyResult { let mut names: Vec = self.schema_names().into_iter().collect(); names.sort(); Ok(format!("Catalog(schema_names=[{}])", names.join(", "))) } } #[pymethods] impl PySchema { #[new] pub fn new( py: Python, schema_provider: Py, session: Option>, ) -> PyResult { let codec = extract_logical_extension_codec(py, session)?; let schema = Arc::new(RustWrappedPySchemaProvider::new(schema_provider)) as Arc; Ok(Self { schema, codec }) } #[staticmethod] fn memory_schema(py: Python, session: Option>) -> PyResult { let codec = extract_logical_extension_codec(py, session)?; let schema = Arc::new(MemorySchemaProvider::default()) as Arc; Ok(Self { schema, codec }) } #[getter] fn table_names(&self) -> HashSet { self.schema.table_names().into_iter().collect() } fn table(&self, name: &str, py: Python) -> PyDataFusionResult { if let Some(table) = wait_for_future(py, self.schema.table(name))?? { Ok(PyTable::from(table)) } else { Err(PyDataFusionError::Common(format!( "Table not found: {name}" ))) } } fn __repr__(&self) -> PyResult { let mut names: Vec = self.table_names().into_iter().collect(); names.sort(); Ok(format!("Schema(table_names=[{}])", names.join(";"))) } fn register_table(&self, name: &str, table_provider: Bound<'_, PyAny>) -> PyResult<()> { let py = table_provider.py(); let codec_capsule = create_logical_extension_capsule(py, self.codec.as_ref())? .as_any() .clone(); let table = PyTable::new(table_provider, Some(codec_capsule))?; let _ = self .schema .register_table(name.to_string(), table.table) .map_err(py_datafusion_err)?; Ok(()) } fn deregister_table(&self, name: &str) -> PyResult<()> { let _ = self .schema .deregister_table(name) .map_err(py_datafusion_err)?; Ok(()) } fn table_exist(&self, name: &str) -> bool { self.schema.table_exist(name) } } #[derive(Debug)] pub(crate) struct RustWrappedPySchemaProvider { schema_provider: Py, owner_name: Option, } impl RustWrappedPySchemaProvider { pub fn new(schema_provider: Py) -> Self { let owner_name = Python::attach(|py| { schema_provider .bind(py) .getattr("owner_name") .ok() .map(|name| name.to_string()) }); Self { schema_provider, owner_name, } } fn table_inner(&self, name: &str) -> PyResult>> { Python::attach(|py| { let provider = self.schema_provider.bind(py); let py_table_method = provider.getattr("table")?; let py_table = py_table_method.call((name,), None)?; if py_table.is_none() { return Ok(None); } let table = PyTable::new(py_table, None)?; Ok(Some(table.table)) }) } } #[async_trait] impl SchemaProvider for RustWrappedPySchemaProvider { fn owner_name(&self) -> Option<&str> { self.owner_name.as_deref() } fn table_names(&self) -> Vec { Python::attach(|py| { let provider = self.schema_provider.bind(py); provider .getattr("table_names") .and_then(|names| names.extract::>()) .unwrap_or_else(|err| { log::error!("Unable to get table_names: {err}"); Vec::default() }) }) } async fn table( &self, name: &str, ) -> datafusion::common::Result>, DataFusionError> { self.table_inner(name) .map_err(|e| DataFusionError::External(Box::new(e))) } fn register_table( &self, name: String, table: Arc, ) -> datafusion::common::Result>> { let py_table = PyTable::from(table); Python::attach(|py| { let provider = self.schema_provider.bind(py); let _ = provider .call_method1("register_table", (name, py_table)) .map_err(to_datafusion_err)?; // Since the definition of `register_table` says that an error // will be returned if the table already exists, there is no // case where we want to return a table provider as output. Ok(None) }) } fn deregister_table( &self, name: &str, ) -> datafusion::common::Result>> { Python::attach(|py| { let provider = self.schema_provider.bind(py); let table = provider .call_method1("deregister_table", (name,)) .map_err(to_datafusion_err)?; if table.is_none() { return Ok(None); } // If we can turn this table provider into a `Dataset`, return it. // Otherwise, return None. let dataset = match Dataset::new(&table, py) { Ok(dataset) => Some(Arc::new(dataset) as Arc), Err(_) => None, }; Ok(dataset) }) } fn table_exist(&self, name: &str) -> bool { Python::attach(|py| { let provider = self.schema_provider.bind(py); provider .call_method1("table_exist", (name,)) .and_then(|pyobj| pyobj.extract()) .unwrap_or(false) }) } } #[derive(Debug)] pub(crate) struct RustWrappedPyCatalogProvider { pub(crate) catalog_provider: Py, codec: Arc, } impl RustWrappedPyCatalogProvider { pub fn new(catalog_provider: Py, codec: Arc) -> Self { Self { catalog_provider, codec, } } fn schema_inner(&self, name: &str) -> PyResult>> { Python::attach(|py| { let provider = self.catalog_provider.bind(py); let py_schema = provider.call_method1("schema", (name,))?; if py_schema.is_none() { return Ok(None); } extract_schema_provider_from_pyobj(py_schema, self.codec.as_ref()).map(Some) }) } } #[async_trait] impl CatalogProvider for RustWrappedPyCatalogProvider { fn schema_names(&self) -> Vec { Python::attach(|py| { let provider = self.catalog_provider.bind(py); provider .call_method0("schema_names") .and_then(|names| names.extract::>()) .map(|names| names.into_iter().collect()) .unwrap_or_else(|err| { log::error!("Unable to get schema_names: {err}"); Vec::default() }) }) } fn schema(&self, name: &str) -> Option> { self.schema_inner(name).unwrap_or_else(|err| { log::error!("CatalogProvider schema returned error: {err}"); None }) } fn register_schema( &self, name: &str, schema: Arc, ) -> datafusion::common::Result>> { Python::attach(|py| { let py_schema = match schema.downcast_ref::() { Some(wrapped_schema) => wrapped_schema.schema_provider.as_any(), None => &PySchema::new_from_parts(schema, self.codec.clone()) .into_py_any(py) .map_err(to_datafusion_err)?, }; let provider = self.catalog_provider.bind(py); let schema = provider .call_method1("register_schema", (name, py_schema)) .map_err(to_datafusion_err)?; if schema.is_none() { return Ok(None); } let schema = Arc::new(RustWrappedPySchemaProvider::new(schema.into())) as Arc; Ok(Some(schema)) }) } fn deregister_schema( &self, name: &str, cascade: bool, ) -> datafusion::common::Result>> { Python::attach(|py| { let provider = self.catalog_provider.bind(py); let schema = provider .call_method1("deregister_schema", (name, cascade)) .map_err(to_datafusion_err)?; if schema.is_none() { return Ok(None); } let schema = Arc::new(RustWrappedPySchemaProvider::new(schema.into())) as Arc; Ok(Some(schema)) }) } } #[derive(Debug)] pub(crate) struct RustWrappedPyCatalogProviderList { pub(crate) catalog_provider_list: Py, codec: Arc, } impl RustWrappedPyCatalogProviderList { pub fn new(catalog_provider_list: Py, codec: Arc) -> Self { Self { catalog_provider_list, codec, } } fn catalog_inner(&self, name: &str) -> PyResult>> { Python::attach(|py| { let provider = self.catalog_provider_list.bind(py); let py_schema = provider.call_method1("catalog", (name,))?; if py_schema.is_none() { return Ok(None); } extract_catalog_provider_from_pyobj(py_schema, self.codec.as_ref()).map(Some) }) } } #[async_trait] impl CatalogProviderList for RustWrappedPyCatalogProviderList { fn catalog_names(&self) -> Vec { Python::attach(|py| { let provider = self.catalog_provider_list.bind(py); provider .call_method0("catalog_names") .and_then(|names| names.extract::>()) .map(|names| names.into_iter().collect()) .unwrap_or_else(|err| { log::error!("Unable to get catalog_names: {err}"); Vec::default() }) }) } fn catalog(&self, name: &str) -> Option> { self.catalog_inner(name).unwrap_or_else(|err| { log::error!("CatalogProvider catalog returned error: {err}"); None }) } fn register_catalog( &self, name: String, catalog: Arc, ) -> Option> { Python::attach(|py| { let py_catalog = match catalog.downcast_ref::() { Some(wrapped_schema) => wrapped_schema.catalog_provider.as_any().clone_ref(py), None => { match PyCatalog::new_from_parts(catalog, self.codec.clone()).into_py_any(py) { Ok(c) => c, Err(err) => { log::error!( "register_catalog returned error during conversion to PyAny: {err}" ); return None; } } } }; let provider = self.catalog_provider_list.bind(py); let catalog = match provider.call_method1("register_catalog", (name, py_catalog)) { Ok(c) => c, Err(err) => { log::error!("register_catalog returned error: {err}"); return None; } }; if catalog.is_none() { return None; } let catalog = Arc::new(RustWrappedPyCatalogProvider::new( catalog.into(), self.codec.clone(), )) as Arc; Some(catalog) }) } } fn extract_catalog_provider_from_pyobj( mut catalog_provider: Bound, codec: &FFI_LogicalExtensionCodec, ) -> PyResult> { if catalog_provider.hasattr("__datafusion_catalog_provider__")? { let py = catalog_provider.py(); let codec_capsule = create_logical_extension_capsule(py, codec)?; catalog_provider = catalog_provider .getattr("__datafusion_catalog_provider__")? .call1((codec_capsule,))?; } let provider = if let Ok(capsule) = catalog_provider.cast::() { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_catalog_provider"))? .cast(); let provider = unsafe { data.as_ref() }; let provider: Arc = provider.into(); provider } else { match catalog_provider.extract::() { Ok(py_catalog) => py_catalog.catalog, Err(_) => Arc::new(RustWrappedPyCatalogProvider::new( catalog_provider.into(), Arc::new(codec.clone()), )) as Arc, } }; Ok(provider) } fn extract_schema_provider_from_pyobj( mut schema_provider: Bound, codec: &FFI_LogicalExtensionCodec, ) -> PyResult> { if schema_provider.hasattr("__datafusion_schema_provider__")? { let py = schema_provider.py(); let codec_capsule = create_logical_extension_capsule(py, codec)?; schema_provider = schema_provider .getattr("__datafusion_schema_provider__")? .call1((codec_capsule,))?; } let provider = if let Ok(capsule) = schema_provider.cast::() { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_schema_provider"))? .cast(); let provider = unsafe { data.as_ref() }; let provider: Arc = provider.into(); provider } else { match schema_provider.extract::() { Ok(py_schema) => py_schema.schema, Err(_) => Arc::new(RustWrappedPySchemaProvider::new(schema_provider.into())) as Arc, } }; Ok(provider) } fn extract_logical_extension_codec( py: Python, obj: Option>, ) -> PyResult> { let obj = match obj { Some(obj) => obj, None => PySessionContext::global_ctx()?.into_bound_py_any(py)?, }; ffi_logical_codec_from_pycapsule(obj).map(Arc::new) } pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/codec.rs000066400000000000000000001325421521727353300231350ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Python-aware extension codecs. //! //! Datafusion-python plans can carry references to Python-defined //! objects that the upstream protobuf codecs do not know how to //! serialize: pure-Python scalar / aggregate / window UDFs, Python //! query-planning extensions, and so on. Their state lives inside //! `Py` callables and closures rather than being recoverable //! from a name in the receiver's function registry. To ship a plan //! across a process boundary (pickle, `multiprocessing`, Ray actor, //! `datafusion-distributed`, etc.) those payloads have to be encoded //! into the proto wire format itself. //! //! [`PythonLogicalCodec`] is the [`LogicalExtensionCodec`] that //! datafusion-python parks on every `SessionContext`. It wraps a //! user-supplied (or default) inner codec and adds Python-aware //! in-band encoding on top: when the encoder sees a Python-defined //! UDF, the codec cloudpickles the callable + signature into the //! `fun_definition` proto field; when the decoder sees a payload it //! produced, it reconstructs the UDF from the bytes alone — no //! pre-registration on the receiver. UDFs the codec does not //! recognise are delegated to `inner`, which is typically //! `DefaultLogicalExtensionCodec` but may be a downstream-supplied //! FFI codec installed via //! `SessionContext.with_logical_extension_codec(...)`. //! //! [`PythonPhysicalCodec`] is the symmetric wrapper around //! [`PhysicalExtensionCodec`]. Logical and physical layers each have //! a `try_encode_udf` / `try_decode_udf` pair, so a `ScalarUDF` //! referenced inside a `LogicalPlan`, an `ExecutionPlan`, or a //! `PhysicalExpr` must encode identically through either layer for //! plans to survive a serialization round-trip. Both codecs share //! the same payload framing for that reason. //! //! Payloads emitted by these codecs are framed as //! ` `. //! The family magic identifies the UDF flavor; the version byte lets //! the decoder reject too-new or too-old payloads with a clean error //! instead of falling into an opaque `cloudpickle` tuple-unpack //! failure when the tuple shape changes; the Python `(major, minor)` //! bytes catch the cloudpickle-cross-minor-version case and raise an //! actionable error instead of an opaque `marshal` failure on load //! (cloudpickle payloads are not portable across Python minor //! versions). Dispatch precedence on decode: **family match + //! supported version + matching Python version → `inner` codec → //! caller's `FunctionRegistry` fallback.** //! //! ## Wire-format family registry //! //! | Layer + kind | Family prefix | //! | ----------------------------- | ------------- | //! | `PythonLogicalCodec` scalar | `DFPYUDF` | //! | `PythonLogicalCodec` agg | `DFPYUDA` | //! | `PythonLogicalCodec` window | `DFPYUDW` | //! | `PythonPhysicalCodec` scalar | `DFPYUDF` | //! | `PythonPhysicalCodec` agg | `DFPYUDA` | //! | `PythonPhysicalCodec` window | `DFPYUDW` | //! | User FFI extension codec | user-chosen | //! | Default codec | (none) | //! //! Current wire-format version is [`WIRE_VERSION_CURRENT`]; supported //! receive range is `WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT`. //! Bump [`WIRE_VERSION_CURRENT`] whenever the cloudpickle tuple shape //! changes; raise [`WIRE_VERSION_MIN_SUPPORTED`] when dropping support //! for an older shape. //! //! Downstream FFI codecs should pick non-colliding family prefixes //! (use a `DF` namespace plus a crate-specific suffix). The codec //! implementations in this module currently delegate every method to //! `inner`; the encoder/decoder hooks for each kind are added as the //! corresponding Python-side type becomes serializable. use std::sync::Arc; use arrow::datatypes::{DataType, Field, Schema, SchemaRef}; use arrow::ipc::reader::StreamReader; use arrow::ipc::writer::StreamWriter; use datafusion::common::{Result, TableReference}; use datafusion::datasource::TableProvider; use datafusion::datasource::file_format::FileFormatFactory; use datafusion::execution::TaskContext; use datafusion::logical_expr::{ AggregateUDF, AggregateUDFImpl, Extension, LogicalPlan, ScalarUDF, ScalarUDFImpl, Signature, TypeSignature, Volatility, WindowUDF, WindowUDFImpl, }; use datafusion::physical_expr::PhysicalExpr; use datafusion::physical_plan::ExecutionPlan; use datafusion_proto::logical_plan::{DefaultLogicalExtensionCodec, LogicalExtensionCodec}; use datafusion_proto::physical_plan::{DefaultPhysicalExtensionCodec, PhysicalExtensionCodec}; use pyo3::prelude::*; use pyo3::sync::PyOnceLock; use pyo3::types::{PyBytes, PyTuple}; use crate::errors::to_datafusion_err; use crate::udaf::PythonFunctionAggregateUDF; use crate::udf::PythonFunctionScalarUDF; use crate::udwf::PythonFunctionWindowUDF; // Wire-format framing for inlined Python UDF payloads. // // Layout: ` `. // The family magic identifies the UDF flavor; the version byte lets // the decoder reject too-new or too-old payloads with a clean error // instead of falling into an opaque `cloudpickle` tuple-unpack failure // when the tuple shape changes; the Python `(major, minor)` bytes // catch the cloudpickle-cross-minor-version case (cloudpickle is not // portable across Python minor versions) and raise an actionable // error instead of an opaque `marshal` failure on load. Bump // [`WIRE_VERSION_CURRENT`] whenever the tuple shape changes; raise // [`WIRE_VERSION_MIN_SUPPORTED`] when dropping support for an older // shape. /// Family prefix for an inlined Python scalar UDF /// (cloudpickled tuple of name, callable, input schema, return field, /// volatility). pub(crate) const PY_SCALAR_UDF_FAMILY: &[u8] = b"DFPYUDF"; /// Family prefix for an inlined Python aggregate UDF /// (cloudpickled tuple of name, accumulator factory, input schema bytes, /// return schema bytes (single-field IPC schema), state schema bytes, /// volatility). pub(crate) const PY_AGG_UDF_FAMILY: &[u8] = b"DFPYUDA"; /// Family prefix for an inlined Python window UDF /// (cloudpickled tuple of name, evaluator factory, input schema bytes, /// return schema bytes (single-field IPC schema), volatility). pub(crate) const PY_WINDOW_UDF_FAMILY: &[u8] = b"DFPYUDW"; /// Wire-format version this build emits. pub(crate) const WIRE_VERSION_CURRENT: u8 = 1; /// Oldest wire-format version this build still decodes. Bump when /// retiring support for an older payload shape. pub(crate) const WIRE_VERSION_MIN_SUPPORTED: u8 = 1; /// Tag `buf` with the framing header for `family` at the current /// wire-format version, stamping `py_version` as `(major, minor)` /// bytes. Append-only — the caller writes the cloudpickle payload /// after. fn write_wire_header(buf: &mut Vec, family: &[u8], py_version: (u8, u8)) { buf.extend_from_slice(family); buf.push(WIRE_VERSION_CURRENT); buf.push(py_version.0); buf.push(py_version.1); } /// Inspect the framing on `buf`. /// /// * `Ok(None)` — `buf` does not carry `family`. The caller should /// delegate to its `inner` codec. /// * `Ok(Some(payload))` — `buf` carries `family` at a version this /// build accepts and a Python `(major, minor)` matching /// `expected_py`; `payload` is the cloudpickle blob. /// * `Err(_)` — `buf` carries `family` but the wire-format version /// is outside `WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT`, /// or the stamped Python `(major, minor)` does not match /// `expected_py`. The error names the offending values so an /// operator can diagnose sender/receiver drift instead of seeing /// an opaque cloudpickle tuple-unpack or `marshal` failure. fn strip_wire_header<'a>( buf: &'a [u8], family: &[u8], kind: &str, expected_py: (u8, u8), ) -> Result> { if !buf.starts_with(family) { return Ok(None); } let version_idx = family.len(); let Some(&version) = buf.get(version_idx) else { return Err(datafusion::error::DataFusionError::Execution(format!( "Truncated inline Python {kind} payload: missing wire-format version byte" ))); }; if !(WIRE_VERSION_MIN_SUPPORTED..=WIRE_VERSION_CURRENT).contains(&version) { return Err(datafusion::error::DataFusionError::Execution(format!( "Inline Python {kind} payload wire-format version v{version}; \ this build supports v{WIRE_VERSION_MIN_SUPPORTED}..=v{WIRE_VERSION_CURRENT}. \ Align datafusion-python versions on sender and receiver." ))); } let py_major_idx = version_idx + 1; let Some(&encoded_major) = buf.get(py_major_idx) else { return Err(datafusion::error::DataFusionError::Execution(format!( "Truncated inline Python {kind} payload: missing Python major version byte" ))); }; let py_minor_idx = version_idx + 2; let Some(&encoded_minor) = buf.get(py_minor_idx) else { return Err(datafusion::error::DataFusionError::Execution(format!( "Truncated inline Python {kind} payload: missing Python minor version byte" ))); }; let (current_major, current_minor) = expected_py; if encoded_major != current_major || encoded_minor != current_minor { return Err(datafusion::error::DataFusionError::Execution(format!( "Inline Python {kind} payload was serialized on Python \ {encoded_major}.{encoded_minor} but this process is running Python \ {current_major}.{current_minor}. cloudpickle payloads are not portable \ across Python minor versions. Align Python versions on sender and receiver." ))); } Ok(Some(&buf[py_minor_idx + 1..])) } /// `LogicalExtensionCodec` parked on every `SessionContext`. Holds /// the Python-aware encoding hooks for logical-layer types /// (`LogicalPlan`, `Expr`) and delegates everything it does not /// handle to the composable `inner` codec — typically /// `DefaultLogicalExtensionCodec`, or a downstream FFI codec /// installed via `SessionContext.with_logical_extension_codec(...)`. /// /// Sitting at the top of the session's logical codec stack means /// every serializer that reads `session.logical_codec()` automatically /// picks up Python-aware encoding for free. #[derive(Debug)] pub struct PythonLogicalCodec { inner: Arc, python_udf_inlining: bool, } impl PythonLogicalCodec { pub fn new(inner: Arc) -> Self { Self { inner, python_udf_inlining: true, } } pub fn inner(&self) -> &Arc { &self.inner } /// Toggle inline encoding of Python UDFs. See /// `SessionContext.with_python_udf_inlining` (Python) for full /// behavior and use cases. /// /// Security scope: strict mode (`false`) narrows only the codec /// layer — it stops `Expr::from_bytes` from invoking /// `cloudpickle.loads` on the inline `DFPY*` payload. It does /// **not** make `pickle.loads(untrusted_bytes)` safe; treat every /// `pickle.loads` on untrusted input as unsafe regardless of this /// setting. See `docs/source/user-guide/io/distributing_work.rst` /// (Security section) for the full threat model, and Python's /// [pickle module security warning][1] for why `pickle.loads` is /// unsafe in general. /// /// [1]: https://docs.python.org/3/library/pickle.html#module-pickle pub fn with_python_udf_inlining(mut self, enabled: bool) -> Self { self.python_udf_inlining = enabled; self } pub fn python_udf_inlining(&self) -> bool { self.python_udf_inlining } } impl Default for PythonLogicalCodec { fn default() -> Self { Self::new(Arc::new(DefaultLogicalExtensionCodec {})) } } impl LogicalExtensionCodec for PythonLogicalCodec { fn try_decode( &self, buf: &[u8], inputs: &[LogicalPlan], ctx: &TaskContext, ) -> Result { self.inner.try_decode(buf, inputs, ctx) } fn try_encode(&self, node: &Extension, buf: &mut Vec) -> Result<()> { self.inner.try_encode(node, buf) } fn try_decode_table_provider( &self, buf: &[u8], table_ref: &TableReference, schema: SchemaRef, ctx: &TaskContext, ) -> Result> { self.inner .try_decode_table_provider(buf, table_ref, schema, ctx) } fn try_encode_table_provider( &self, table_ref: &TableReference, node: Arc, buf: &mut Vec, ) -> Result<()> { self.inner.try_encode_table_provider(table_ref, node, buf) } fn try_decode_file_format( &self, buf: &[u8], ctx: &TaskContext, ) -> Result> { self.inner.try_decode_file_format(buf, ctx) } fn try_encode_file_format( &self, buf: &mut Vec, node: Arc, ) -> Result<()> { self.inner.try_encode_file_format(buf, node) } fn try_encode_udf(&self, node: &ScalarUDF, buf: &mut Vec) -> Result<()> { if self.python_udf_inlining && try_encode_python_scalar_udf(node, buf)? { return Ok(()); } self.inner.try_encode_udf(node, buf) } fn try_decode_udf(&self, name: &str, buf: &[u8]) -> Result> { if self.python_udf_inlining { if let Some(udf) = try_decode_python_scalar_udf(buf)? { return Ok(udf); } } else { refuse_if_inline(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", name)?; } self.inner.try_decode_udf(name, buf) } fn try_encode_udaf(&self, node: &AggregateUDF, buf: &mut Vec) -> Result<()> { if self.python_udf_inlining && try_encode_python_udaf(node, buf)? { return Ok(()); } self.inner.try_encode_udaf(node, buf) } fn try_decode_udaf(&self, name: &str, buf: &[u8]) -> Result> { if self.python_udf_inlining { if let Some(udaf) = try_decode_python_udaf(buf)? { return Ok(udaf); } } else { refuse_if_inline(buf, PY_AGG_UDF_FAMILY, "aggregate UDF", name)?; } self.inner.try_decode_udaf(name, buf) } fn try_encode_udwf(&self, node: &WindowUDF, buf: &mut Vec) -> Result<()> { if self.python_udf_inlining && try_encode_python_udwf(node, buf)? { return Ok(()); } self.inner.try_encode_udwf(node, buf) } fn try_decode_udwf(&self, name: &str, buf: &[u8]) -> Result> { if self.python_udf_inlining { if let Some(udwf) = try_decode_python_udwf(buf)? { return Ok(udwf); } } else { refuse_if_inline(buf, PY_WINDOW_UDF_FAMILY, "window UDF", name)?; } self.inner.try_decode_udwf(name, buf) } } /// Strict-mode gate: if `buf` is a well-framed inline payload for /// `family`, return the strict-refusal error; otherwise return /// `Ok(())` so the caller can delegate to its `inner` codec. /// /// Routing through [`read_framed_payload`] (rather than a bare /// `starts_with` probe) means malformed inline bytes — wrong /// wire-format version, mismatched Python version, truncated header — /// surface *their* diagnostic instead of the strict-mode message. /// The strict message implies sender intent ("inlining is disabled"), /// so it should fire only when the bytes really would have decoded. /// /// Fast path: short-circuit on the family-magic prefix before /// acquiring the GIL. Plans with many non-Python UDFs would otherwise /// pay a GIL acquisition per decode call just to confirm "not a /// Python UDF". `read_framed_payload` itself rejects buffers that /// don't start with `family`, so this is purely an optimization. fn refuse_if_inline(buf: &[u8], family: &[u8], kind: &str, name: &str) -> Result<()> { if !buf.starts_with(family) { return Ok(()); } Python::attach(|py| match read_framed_payload(py, buf, family, kind)? { Some(_) => Err(refuse_inline_payload(kind, name)), None => Ok(()), }) } /// Build the error returned by a strict codec when it receives an /// inline Python-UDF payload it has been told not to deserialize. fn refuse_inline_payload(kind: &str, name: &str) -> datafusion::error::DataFusionError { // `Execution`, not `Plan`: this is a wire-format decode refusal at // codec time, not a planner-stage failure. Downstream error // classification keys off the variant — surfacing this as a planner // error would mis-route it into "fix your SQL" buckets. datafusion::error::DataFusionError::Execution(format!( "Refusing to deserialize inline Python {kind} '{name}': Python UDF \ inlining is disabled on this session. Two remediations: \ (1) ask the sender to re-encode with inlining disabled so '{name}' \ travels by name, and register '{name}' on this receiver; or \ (2) enable inlining on this receiver (accepts the cloudpickle \ execution risk on inbound payloads). Receivers cannot re-encode \ bytes they did not produce." )) } /// `PhysicalExtensionCodec` mirror of [`PythonLogicalCodec`] parked /// on the same `SessionContext`. Carries the Python-aware encoding /// hooks for physical-layer types (`ExecutionPlan`, `PhysicalExpr`) /// and delegates the rest to `inner`. /// /// The `PhysicalExtensionCodec` trait has its own `try_encode_udf` /// / `try_decode_udf` pair distinct from the logical one, so a /// `ScalarUDF` referenced inside a physical plan needs Python-aware /// encoding on this layer too — otherwise a plan with a Python UDF /// would round-trip at the logical level but break at the physical /// level. Both layers reuse the shared payload framing /// ([`PY_SCALAR_UDF_FAMILY`] et al.) so the wire format is identical. #[derive(Debug)] pub struct PythonPhysicalCodec { inner: Arc, python_udf_inlining: bool, } impl PythonPhysicalCodec { pub fn new(inner: Arc) -> Self { Self { inner, python_udf_inlining: true, } } pub fn inner(&self) -> &Arc { &self.inner } /// Toggle inline encoding of Python UDFs on this physical codec. /// /// Mirrors [`PythonLogicalCodec::with_python_udf_inlining`]; see /// that method for the full security and portability discussion. pub fn with_python_udf_inlining(mut self, enabled: bool) -> Self { self.python_udf_inlining = enabled; self } pub fn python_udf_inlining(&self) -> bool { self.python_udf_inlining } } impl Default for PythonPhysicalCodec { fn default() -> Self { Self::new(Arc::new(DefaultPhysicalExtensionCodec {})) } } impl PhysicalExtensionCodec for PythonPhysicalCodec { fn try_decode( &self, buf: &[u8], inputs: &[Arc], ctx: &TaskContext, ) -> Result> { self.inner.try_decode(buf, inputs, ctx) } fn try_encode(&self, node: Arc, buf: &mut Vec) -> Result<()> { self.inner.try_encode(node, buf) } fn try_encode_udf(&self, node: &ScalarUDF, buf: &mut Vec) -> Result<()> { if self.python_udf_inlining && try_encode_python_scalar_udf(node, buf)? { return Ok(()); } self.inner.try_encode_udf(node, buf) } fn try_decode_udf(&self, name: &str, buf: &[u8]) -> Result> { if self.python_udf_inlining { if let Some(udf) = try_decode_python_scalar_udf(buf)? { return Ok(udf); } } else { refuse_if_inline(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", name)?; } self.inner.try_decode_udf(name, buf) } fn try_encode_expr(&self, node: &Arc, buf: &mut Vec) -> Result<()> { self.inner.try_encode_expr(node, buf) } fn try_decode_expr( &self, buf: &[u8], inputs: &[Arc], ) -> Result> { self.inner.try_decode_expr(buf, inputs) } fn try_encode_udaf(&self, node: &AggregateUDF, buf: &mut Vec) -> Result<()> { if self.python_udf_inlining && try_encode_python_udaf(node, buf)? { return Ok(()); } self.inner.try_encode_udaf(node, buf) } fn try_decode_udaf(&self, name: &str, buf: &[u8]) -> Result> { if self.python_udf_inlining { if let Some(udaf) = try_decode_python_udaf(buf)? { return Ok(udaf); } } else { refuse_if_inline(buf, PY_AGG_UDF_FAMILY, "aggregate UDF", name)?; } self.inner.try_decode_udaf(name, buf) } fn try_encode_udwf(&self, node: &WindowUDF, buf: &mut Vec) -> Result<()> { if self.python_udf_inlining && try_encode_python_udwf(node, buf)? { return Ok(()); } self.inner.try_encode_udwf(node, buf) } fn try_decode_udwf(&self, name: &str, buf: &[u8]) -> Result> { if self.python_udf_inlining { if let Some(udwf) = try_decode_python_udwf(buf)? { return Ok(udwf); } } else { refuse_if_inline(buf, PY_WINDOW_UDF_FAMILY, "window UDF", name)?; } self.inner.try_decode_udwf(name, buf) } } // ============================================================================= // Shared Python scalar UDF encode / decode helpers // // Both `PythonLogicalCodec` and `PythonPhysicalCodec` consult these on // every `try_encode_udf` / `try_decode_udf` call. Same wire format on // both layers — a Python `ScalarUDF` referenced inside a `LogicalPlan` // or an `ExecutionPlan` round-trips identically. // ============================================================================= /// Encode a Python scalar UDF inline if `node` is one. Returns /// `Ok(true)` when the payload (`DFPYUDF` family prefix, version byte, /// cloudpickled tuple) was written and the caller should skip its /// inner codec. Returns `Ok(false)` for any non-Python UDF, signalling /// the caller to delegate to its `inner`. pub(crate) fn try_encode_python_scalar_udf(node: &ScalarUDF, buf: &mut Vec) -> Result { let Some(py_udf) = node.inner().downcast_ref::() else { return Ok(false); }; Python::attach(|py| -> Result { let bytes = encode_python_scalar_udf(py, py_udf).map_err(to_datafusion_err)?; append_framed_payload(py, buf, PY_SCALAR_UDF_FAMILY, &bytes)?; Ok(true) }) } /// Decode an inline Python scalar UDF payload. Returns `Ok(None)` /// when `buf` does not carry the `DFPYUDF` family prefix, signalling /// the caller to delegate to its `inner` codec (and eventually the /// `FunctionRegistry`). pub(crate) fn try_decode_python_scalar_udf(buf: &[u8]) -> Result>> { if !buf.starts_with(PY_SCALAR_UDF_FAMILY) { return Ok(None); } Python::attach(|py| -> Result>> { let Some(payload) = read_framed_payload(py, buf, PY_SCALAR_UDF_FAMILY, "scalar UDF")? else { return Ok(None); }; let udf = decode_python_scalar_udf(py, payload).map_err(to_datafusion_err)?; Ok(Some(Arc::new(ScalarUDF::new_from_impl(udf)))) }) } /// Build the cloudpickle payload for a `PythonFunctionScalarUDF`. /// /// Layout: `cloudpickle.dumps((name, func, input_schema_bytes, /// return_schema_bytes, volatility_str))`. Schema blobs are produced /// by arrow-rs's native IPC stream writer (no pyarrow round-trip) and /// decoded with the matching stream reader on the receiver. See /// [`build_input_schema_bytes`] for what the input blob carries. fn encode_python_scalar_udf(py: Python<'_>, udf: &PythonFunctionScalarUDF) -> PyResult> { let signature = udf.signature(); let input_dtypes = signature_input_dtypes(signature, "PythonFunctionScalarUDF")?; let input_schema_bytes = build_input_schema_bytes(&input_dtypes)?; let return_schema_bytes = build_single_field_schema_bytes(udf.return_field().as_ref())?; let volatility = volatility_wire_str(signature.volatility); let payload = PyTuple::new( py, [ udf.name().into_pyobject(py)?.into_any(), udf.func().bind(py).clone().into_any(), PyBytes::new(py, &input_schema_bytes).into_any(), PyBytes::new(py, &return_schema_bytes).into_any(), volatility.into_pyobject(py)?.into_any(), ], )?; cloudpickle(py)? .call_method1("dumps", (payload,))? .extract::>() } /// Inverse of [`encode_python_scalar_udf`]. fn decode_python_scalar_udf(py: Python<'_>, payload: &[u8]) -> PyResult { let tuple = cloudpickle(py)? .call_method1("loads", (PyBytes::new(py, payload),))? .cast_into::()?; let name: String = tuple.get_item(0)?.extract()?; let func: Py = tuple.get_item(1)?.unbind(); let input_schema_bytes: Vec = tuple.get_item(2)?.extract()?; let return_schema_bytes: Vec = tuple.get_item(3)?.extract()?; let volatility_str: String = tuple.get_item(4)?.extract()?; let input_types = read_input_dtypes(&input_schema_bytes)?; let return_field = read_single_return_field(&return_schema_bytes, "PythonFunctionScalarUDF")?; let volatility = parse_volatility_str(&volatility_str)?; Ok(PythonFunctionScalarUDF::from_parts( name, func, input_types, return_field, volatility, )) } /// Serialize a `Schema` to a self-contained IPC stream containing /// only the schema message (no record batches). Inverse: /// [`schema_from_ipc_bytes`]. fn schema_to_ipc_bytes(schema: &Schema) -> arrow::error::Result> { let mut buf: Vec = Vec::new(); { let mut writer = StreamWriter::try_new(&mut buf, schema)?; writer.finish()?; } Ok(buf) } /// Decode an IPC stream containing only a schema message back into a /// `Schema`. Inverse: [`schema_to_ipc_bytes`]. fn schema_from_ipc_bytes(bytes: &[u8]) -> arrow::error::Result { let reader = StreamReader::try_new(std::io::Cursor::new(bytes), None)?; Ok(reader.schema().as_ref().clone()) } /// Extract the per-arg `DataType`s from a `Signature` known to be /// `TypeSignature::Exact` (all Python-defined UDFs are constructed /// with `Signature::exact`). Any other variant indicates the impl was /// not built by this crate's UDF/UDAF/UDWF constructors. fn signature_input_dtypes(signature: &Signature, kind: &str) -> PyResult> { match &signature.type_signature { TypeSignature::Exact(types) => Ok(types.clone()), other => Err(pyo3::exceptions::PyValueError::new_err(format!( "{kind} expected Signature::Exact, got {other:?}" ))), } } /// Wrap per-arg `DataType`s in synthetic `arg_{i}` fields and emit /// the IPC schema blob the encoder writes into the cloudpickle tuple. /// /// The names and `nullable: true` are arbitrary: the underlying /// `TypeSignature::Exact` carries no per-input nullability or /// metadata, and the receiver collapses these fields back to /// `Vec` via [`read_input_dtypes`], so anything set here /// beyond the data type is discarded on decode. fn build_input_schema_bytes(dtypes: &[DataType]) -> PyResult> { let fields: Vec = dtypes .iter() .enumerate() .map(|(i, dt)| Field::new(format!("arg_{i}"), dt.clone(), true)) .collect(); schema_to_ipc_bytes(&Schema::new(fields)).map_err(arrow_to_py_err) } /// Emit a single-field IPC schema blob. Used for return-type and /// state-field payloads where the receiver needs to recover field /// metadata (names, nullability, key/value attributes) verbatim. fn build_single_field_schema_bytes(field: &Field) -> PyResult> { schema_to_ipc_bytes(&Schema::new(vec![field.clone()])).map_err(arrow_to_py_err) } /// Emit a multi-field IPC schema blob. fn build_schema_bytes(fields: Vec) -> PyResult> { schema_to_ipc_bytes(&Schema::new(fields)).map_err(arrow_to_py_err) } /// Decode the per-arg `DataType`s the encoder wrote via /// [`build_input_schema_bytes`]. fn read_input_dtypes(bytes: &[u8]) -> PyResult> { let schema = schema_from_ipc_bytes(bytes).map_err(arrow_to_py_err)?; Ok(schema .fields() .iter() .map(|f| f.data_type().clone()) .collect()) } /// Decode a single-field IPC schema blob and return that field by /// value. `kind` names the UDF flavor in the error message produced /// when the blob is empty (should be unreachable for sender-side /// payloads built via [`build_single_field_schema_bytes`]). fn read_single_return_field(bytes: &[u8], kind: &str) -> PyResult { let schema = schema_from_ipc_bytes(bytes).map_err(arrow_to_py_err)?; let field = schema.fields().first().ok_or_else(|| { pyo3::exceptions::PyValueError::new_err(format!( "{kind} return schema must contain exactly one field" )) })?; Ok(field.as_ref().clone()) } fn arrow_to_py_err(e: arrow::error::ArrowError) -> PyErr { pyo3::exceptions::PyValueError::new_err(format!("{e}")) } fn parse_volatility_str(s: &str) -> PyResult { datafusion_python_util::parse_volatility(s) .map_err(|e| pyo3::exceptions::PyValueError::new_err(format!("{e}"))) } /// Stable wire-format string for a `Volatility`. Pinned to the three /// tokens [`datafusion_python_util::parse_volatility`] accepts, so an /// upstream change to `Volatility`'s `Debug` repr cannot silently /// produce bytes the decoder rejects. fn volatility_wire_str(v: Volatility) -> &'static str { match v { Volatility::Immutable => "immutable", Volatility::Stable => "stable", Volatility::Volatile => "volatile", } } /// Read the interpreter's `sys.version_info` as `(major, minor)`. /// /// Used by encoder/decoder to stamp and verify the Python version a /// cloudpickle payload was produced on. cloudpickle is not portable /// across Python minor versions; the wire header carries these bytes /// so a mismatch surfaces an actionable error instead of an opaque /// `marshal` failure at `cloudpickle.loads` time. fn current_python_version(py: Python<'_>) -> PyResult<(u8, u8)> { let version_info = py.import("sys")?.getattr("version_info")?; let major: u8 = version_info.getattr("major")?.extract()?; let minor: u8 = version_info.getattr("minor")?.extract()?; Ok((major, minor)) } /// Stamp `buf` with the framing header for `family` plus the current /// Python `(major, minor)`, then append `payload`. Bundles the /// `current_python_version` lookup with the header write so each /// encoder call site stays one line. fn append_framed_payload( py: Python<'_>, buf: &mut Vec, family: &[u8], payload: &[u8], ) -> Result<()> { let py_version = current_python_version(py).map_err(to_datafusion_err)?; write_wire_header(buf, family, py_version); buf.extend_from_slice(payload); Ok(()) } /// Inspect `buf`'s framing against `family` + the current Python /// `(major, minor)`. Returns `Ok(None)` when `buf` does not carry /// `family` (caller should delegate); `Ok(Some(payload))` when the /// framing matches; `Err(_)` for a recognised family at the wrong /// wire-format or Python version (see [`strip_wire_header`]). fn read_framed_payload<'a>( py: Python<'_>, buf: &'a [u8], family: &[u8], kind: &str, ) -> Result> { let py_version = current_python_version(py).map_err(to_datafusion_err)?; strip_wire_header(buf, family, kind, py_version) } /// Cached handle to the `cloudpickle` module. /// /// The encode/decode helpers above would otherwise re-resolve the /// module on every call. `py.import` is backed by `sys.modules` and /// therefore cheap, but each call still walks a dict and re-binds the /// result; a plan with many Python UDFs pays that cost per UDF. /// /// `PyOnceLock` scopes the cached `Py` to the current /// interpreter, so the slot drops cleanly on interpreter teardown /// (relevant under CPython subinterpreters, PEP 684) instead of /// resurrecting a `Py` rooted in a dead interpreter on the next call. fn cloudpickle<'py>(py: Python<'py>) -> PyResult> { static CLOUDPICKLE: PyOnceLock> = PyOnceLock::new(); CLOUDPICKLE .get_or_try_init(py, || Ok(py.import("cloudpickle")?.unbind().into_any())) .map(|cached| cached.bind(py).clone()) } // ============================================================================= // Shared Python window UDF encode / decode helpers // // Cloudpickle tuple shape: `(name, evaluator_factory, input_schema_bytes, // return_schema_bytes, volatility_str)`. The evaluator factory is the // Python callable that produces a new evaluator instance per partition. // ============================================================================= pub(crate) fn try_encode_python_udwf(node: &WindowUDF, buf: &mut Vec) -> Result { let Some(py_udf) = node.inner().downcast_ref::() else { return Ok(false); }; Python::attach(|py| -> Result { let bytes = encode_python_udwf(py, py_udf).map_err(to_datafusion_err)?; append_framed_payload(py, buf, PY_WINDOW_UDF_FAMILY, &bytes)?; Ok(true) }) } pub(crate) fn try_decode_python_udwf(buf: &[u8]) -> Result>> { if !buf.starts_with(PY_WINDOW_UDF_FAMILY) { return Ok(None); } Python::attach(|py| -> Result>> { let Some(payload) = read_framed_payload(py, buf, PY_WINDOW_UDF_FAMILY, "window UDF")? else { return Ok(None); }; let udf = decode_python_udwf(py, payload).map_err(to_datafusion_err)?; Ok(Some(Arc::new(WindowUDF::new_from_impl(udf)))) }) } fn encode_python_udwf(py: Python<'_>, udf: &PythonFunctionWindowUDF) -> PyResult> { let signature = WindowUDFImpl::signature(udf); let input_dtypes = signature_input_dtypes(signature, "PythonFunctionWindowUDF")?; let input_schema_bytes = build_input_schema_bytes(&input_dtypes)?; let return_field = Field::new("result", udf.return_type().clone(), true); let return_schema_bytes = build_single_field_schema_bytes(&return_field)?; let volatility = volatility_wire_str(signature.volatility); let payload = PyTuple::new( py, [ WindowUDFImpl::name(udf).into_pyobject(py)?.into_any(), udf.evaluator().bind(py).clone().into_any(), PyBytes::new(py, &input_schema_bytes).into_any(), PyBytes::new(py, &return_schema_bytes).into_any(), volatility.into_pyobject(py)?.into_any(), ], )?; cloudpickle(py)? .call_method1("dumps", (payload,))? .extract::>() } fn decode_python_udwf(py: Python<'_>, payload: &[u8]) -> PyResult { let tuple = cloudpickle(py)? .call_method1("loads", (PyBytes::new(py, payload),))? .cast_into::()?; let name: String = tuple.get_item(0)?.extract()?; let evaluator: Py = tuple.get_item(1)?.unbind(); let input_schema_bytes: Vec = tuple.get_item(2)?.extract()?; let return_schema_bytes: Vec = tuple.get_item(3)?.extract()?; let volatility_str: String = tuple.get_item(4)?.extract()?; let input_types = read_input_dtypes(&input_schema_bytes)?; let return_type = read_single_return_field(&return_schema_bytes, "PythonFunctionWindowUDF")? .data_type() .clone(); let volatility = parse_volatility_str(&volatility_str)?; Ok(PythonFunctionWindowUDF::new( name, evaluator, input_types, return_type, volatility, )) } // ============================================================================= // Shared Python aggregate UDF encode / decode helpers // // Cloudpickle tuple shape: `(name, accumulator_factory, input_schema_bytes, // return_schema_bytes, state_schema_bytes, volatility_str)`. The accumulator // factory is the Python callable that produces a new accumulator instance // per partition. // ============================================================================= pub(crate) fn try_encode_python_udaf(node: &AggregateUDF, buf: &mut Vec) -> Result { let Some(py_udf) = node.inner().downcast_ref::() else { return Ok(false); }; Python::attach(|py| -> Result { let bytes = encode_python_udaf(py, py_udf).map_err(to_datafusion_err)?; append_framed_payload(py, buf, PY_AGG_UDF_FAMILY, &bytes)?; Ok(true) }) } pub(crate) fn try_decode_python_udaf(buf: &[u8]) -> Result>> { if !buf.starts_with(PY_AGG_UDF_FAMILY) { return Ok(None); } Python::attach(|py| -> Result>> { let Some(payload) = read_framed_payload(py, buf, PY_AGG_UDF_FAMILY, "aggregate UDF")? else { return Ok(None); }; let udf = decode_python_udaf(py, payload).map_err(to_datafusion_err)?; Ok(Some(Arc::new(AggregateUDF::new_from_impl(udf)))) }) } fn encode_python_udaf(py: Python<'_>, udf: &PythonFunctionAggregateUDF) -> PyResult> { let signature = AggregateUDFImpl::signature(udf); let input_dtypes = signature_input_dtypes(signature, "PythonFunctionAggregateUDF")?; let input_schema_bytes = build_input_schema_bytes(&input_dtypes)?; let return_field = Field::new("result", udf.return_type().clone(), true); let return_schema_bytes = build_single_field_schema_bytes(&return_field)?; let state_fields: Vec = udf .state_fields_ref() .iter() .map(|f| f.as_ref().clone()) .collect(); let state_schema_bytes = build_schema_bytes(state_fields)?; let volatility = volatility_wire_str(signature.volatility); let payload = PyTuple::new( py, [ AggregateUDFImpl::name(udf).into_pyobject(py)?.into_any(), udf.accumulator().bind(py).clone().into_any(), PyBytes::new(py, &input_schema_bytes).into_any(), PyBytes::new(py, &return_schema_bytes).into_any(), PyBytes::new(py, &state_schema_bytes).into_any(), volatility.into_pyobject(py)?.into_any(), ], )?; cloudpickle(py)? .call_method1("dumps", (payload,))? .extract::>() } fn decode_python_udaf(py: Python<'_>, payload: &[u8]) -> PyResult { let tuple = cloudpickle(py)? .call_method1("loads", (PyBytes::new(py, payload),))? .cast_into::()?; let name: String = tuple.get_item(0)?.extract()?; let accumulator: Py = tuple.get_item(1)?.unbind(); let input_schema_bytes: Vec = tuple.get_item(2)?.extract()?; let return_schema_bytes: Vec = tuple.get_item(3)?.extract()?; let state_schema_bytes: Vec = tuple.get_item(4)?.extract()?; let volatility_str: String = tuple.get_item(5)?.extract()?; let input_types = read_input_dtypes(&input_schema_bytes)?; let return_type = read_single_return_field(&return_schema_bytes, "PythonFunctionAggregateUDF")? .data_type() .clone(); // Preserve the encoded state field metadata (names, nullability, // arbitrary key/value attributes) so the post-decode UDF reports // the same state schema as the sender's instance — important for // accumulators whose `StateFieldsArgs` consumers key off names or // nullability rather than positional `DataType`. let state_schema = schema_from_ipc_bytes(&state_schema_bytes).map_err(arrow_to_py_err)?; let state_fields: Vec = state_schema.fields().iter().cloned().collect(); let volatility = parse_volatility_str(&volatility_str)?; Ok(PythonFunctionAggregateUDF::from_parts( name, accumulator, input_types, return_type, state_fields, volatility, )) } #[cfg(test)] mod wire_header_tests { use super::*; const TEST_PY: (u8, u8) = (3, 12); #[test] fn strip_returns_none_when_family_absent() { let buf = b"OTHER_PAYLOAD"; assert!(matches!( strip_wire_header(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY), Ok(None) )); } #[test] fn strip_errors_on_truncated_version_byte() { let buf = PY_SCALAR_UDF_FAMILY; let err = strip_wire_header(buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); assert!(format!("{err}").contains("missing wire-format version byte")); } #[test] fn strip_errors_on_too_new_version() { let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); buf.push(WIRE_VERSION_CURRENT.saturating_add(1)); buf.push(TEST_PY.0); buf.push(TEST_PY.1); buf.extend_from_slice(b"payload"); let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); let msg = format!("{err}"); assert!(msg.contains("wire-format version v")); assert!(msg.contains("supports")); assert!(msg.contains("Align datafusion-python versions")); } #[test] fn strip_errors_on_too_old_version() { if WIRE_VERSION_MIN_SUPPORTED == 0 { return; } let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); buf.push(WIRE_VERSION_MIN_SUPPORTED - 1); buf.push(TEST_PY.0); buf.push(TEST_PY.1); buf.extend_from_slice(b"payload"); assert!(strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).is_err()); } #[test] fn strip_errors_on_truncated_py_major() { let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); buf.push(WIRE_VERSION_CURRENT); let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); assert!(format!("{err}").contains("missing Python major version byte")); } #[test] fn strip_errors_on_truncated_py_minor() { let mut buf = PY_SCALAR_UDF_FAMILY.to_vec(); buf.push(WIRE_VERSION_CURRENT); buf.push(TEST_PY.0); let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY).unwrap_err(); assert!(format!("{err}").contains("missing Python minor version byte")); } #[test] fn strip_errors_on_py_minor_mismatch() { let mut buf = Vec::new(); write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, (3, 11)); buf.extend_from_slice(b"payload"); let err = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", (3, 12)).unwrap_err(); let msg = format!("{err}"); assert!(msg.contains("Python 3.11")); assert!(msg.contains("Python 3.12")); assert!(msg.contains("not portable across Python minor versions")); } #[test] fn strip_errors_on_py_major_mismatch() { let mut buf = Vec::new(); write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, (3, 12)); buf.extend_from_slice(b"payload"); assert!(strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", (4, 0)).is_err()); } #[test] fn write_then_strip_round_trips_scalar_payload() { let mut buf = Vec::new(); write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, TEST_PY); buf.extend_from_slice(b"scalar-payload"); let payload = strip_wire_header(&buf, PY_SCALAR_UDF_FAMILY, "scalar UDF", TEST_PY) .unwrap() .unwrap(); assert_eq!(payload, b"scalar-payload"); } #[test] fn write_then_strip_round_trips_agg_payload() { let mut buf = Vec::new(); write_wire_header(&mut buf, PY_AGG_UDF_FAMILY, TEST_PY); buf.extend_from_slice(b"agg-payload"); let payload = strip_wire_header(&buf, PY_AGG_UDF_FAMILY, "aggregate UDF", TEST_PY) .unwrap() .unwrap(); assert_eq!(payload, b"agg-payload"); } #[test] fn write_then_strip_round_trips_window_payload() { let mut buf = Vec::new(); write_wire_header(&mut buf, PY_WINDOW_UDF_FAMILY, TEST_PY); buf.extend_from_slice(b"window-payload"); let payload = strip_wire_header(&buf, PY_WINDOW_UDF_FAMILY, "window UDF", TEST_PY) .unwrap() .unwrap(); assert_eq!(payload, b"window-payload"); } #[test] fn strip_does_not_match_a_different_family() { let mut buf = Vec::new(); write_wire_header(&mut buf, PY_SCALAR_UDF_FAMILY, TEST_PY); buf.extend_from_slice(b"payload"); assert!(matches!( strip_wire_header(&buf, PY_WINDOW_UDF_FAMILY, "window UDF", TEST_PY), Ok(None) )); } } apache-datafusion-python-54.0.0/crates/core/src/common.rs000066400000000000000000000033721521727353300233460ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use pyo3::prelude::*; pub mod data_type; pub mod df_schema; pub mod function; pub mod schema; /// Initializes the `common` module to match the pattern of `datafusion-common` https://docs.rs/datafusion-common/18.0.0/datafusion_common/index.html pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/common/000077500000000000000000000000001521727353300227735ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/src/common/data_type.rs000066400000000000000000000753741521727353300253330ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::arrow::array::Array; use datafusion::arrow::datatypes::{DataType, IntervalUnit, TimeUnit}; use datafusion::common::ScalarValue; use datafusion::logical_expr::expr::NullTreatment as DFNullTreatment; use pyo3::exceptions::{PyNotImplementedError, PyValueError}; use pyo3::prelude::*; /// A [`ScalarValue`] wrapped in a Python object. This struct allows for conversion /// from a variety of Python objects into a [`ScalarValue`]. See /// ``FromPyArrow::from_pyarrow_bound`` conversion details. #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd)] pub struct PyScalarValue(pub ScalarValue); impl From for PyScalarValue { fn from(value: ScalarValue) -> Self { Self(value) } } impl From for ScalarValue { fn from(value: PyScalarValue) -> Self { value.0 } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "RexType", module = "datafusion.common" )] pub enum RexType { Alias, Literal, Call, Reference, ScalarSubquery, Other, } /// These bindings are tying together several disparate systems. /// You have SQL types for the SQL strings and RDBMS systems itself. /// Rust types for the DataFusion code /// Arrow types which represents the underlying arrow format /// Python types which represent the type in Python /// It is important to keep all of those types in a single /// and manageable location. Therefore this structure exists /// to map those types and provide a simple place for developers /// to map types from one system to another. // TODO: This looks like this needs pyo3 tracking so leaving unfrozen for now #[derive(Debug, Clone)] #[pyclass( from_py_object, name = "DataTypeMap", module = "datafusion.common", subclass )] pub struct DataTypeMap { #[pyo3(get, set)] pub arrow_type: PyDataType, #[pyo3(get, set)] pub python_type: PythonType, #[pyo3(get, set)] pub sql_type: SqlType, } impl DataTypeMap { fn new(arrow_type: DataType, python_type: PythonType, sql_type: SqlType) -> Self { DataTypeMap { arrow_type: PyDataType { data_type: arrow_type, }, python_type, sql_type, } } pub fn map_from_arrow_type(arrow_type: &DataType) -> Result { match arrow_type { DataType::Null => Ok(DataTypeMap::new( DataType::Null, PythonType::None, SqlType::NULL, )), DataType::Boolean => Ok(DataTypeMap::new( DataType::Boolean, PythonType::Bool, SqlType::BOOLEAN, )), DataType::Int8 => Ok(DataTypeMap::new( DataType::Int8, PythonType::Int, SqlType::TINYINT, )), DataType::Int16 => Ok(DataTypeMap::new( DataType::Int16, PythonType::Int, SqlType::SMALLINT, )), DataType::Int32 => Ok(DataTypeMap::new( DataType::Int32, PythonType::Int, SqlType::INTEGER, )), DataType::Int64 => Ok(DataTypeMap::new( DataType::Int64, PythonType::Int, SqlType::BIGINT, )), DataType::UInt8 => Ok(DataTypeMap::new( DataType::UInt8, PythonType::Int, SqlType::TINYINT, )), DataType::UInt16 => Ok(DataTypeMap::new( DataType::UInt16, PythonType::Int, SqlType::SMALLINT, )), DataType::UInt32 => Ok(DataTypeMap::new( DataType::UInt32, PythonType::Int, SqlType::INTEGER, )), DataType::UInt64 => Ok(DataTypeMap::new( DataType::UInt64, PythonType::Int, SqlType::BIGINT, )), DataType::Float16 => Ok(DataTypeMap::new( DataType::Float16, PythonType::Float, SqlType::FLOAT, )), DataType::Float32 => Ok(DataTypeMap::new( DataType::Float32, PythonType::Float, SqlType::FLOAT, )), DataType::Float64 => Ok(DataTypeMap::new( DataType::Float64, PythonType::Float, SqlType::FLOAT, )), DataType::Timestamp(unit, tz) => Ok(DataTypeMap::new( DataType::Timestamp(*unit, tz.clone()), PythonType::Datetime, SqlType::DATE, )), DataType::Date32 => Ok(DataTypeMap::new( DataType::Date32, PythonType::Datetime, SqlType::DATE, )), DataType::Date64 => Ok(DataTypeMap::new( DataType::Date64, PythonType::Datetime, SqlType::DATE, )), DataType::Time32(unit) => Ok(DataTypeMap::new( DataType::Time32(*unit), PythonType::Datetime, SqlType::DATE, )), DataType::Time64(unit) => Ok(DataTypeMap::new( DataType::Time64(*unit), PythonType::Datetime, SqlType::DATE, )), DataType::Duration(_) => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::Interval(interval_unit) => Ok(DataTypeMap::new( DataType::Interval(*interval_unit), PythonType::Datetime, match interval_unit { IntervalUnit::DayTime => SqlType::INTERVAL_DAY, IntervalUnit::MonthDayNano => SqlType::INTERVAL_MONTH, IntervalUnit::YearMonth => SqlType::INTERVAL_YEAR_MONTH, }, )), DataType::Binary => Ok(DataTypeMap::new( DataType::Binary, PythonType::Bytes, SqlType::BINARY, )), DataType::FixedSizeBinary(_) => { Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))) } DataType::LargeBinary => Ok(DataTypeMap::new( DataType::LargeBinary, PythonType::Bytes, SqlType::BINARY, )), DataType::Utf8 => Ok(DataTypeMap::new( DataType::Utf8, PythonType::Str, SqlType::VARCHAR, )), DataType::LargeUtf8 => Ok(DataTypeMap::new( DataType::LargeUtf8, PythonType::Str, SqlType::VARCHAR, )), DataType::List(_) => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::FixedSizeList(_, _) => { Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))) } DataType::LargeList(_) => { Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))) } DataType::Struct(_) => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::Union(_, _) => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::Dictionary(_, _) => { Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))) } DataType::Decimal32(precision, scale) => Ok(DataTypeMap::new( DataType::Decimal32(*precision, *scale), PythonType::Float, SqlType::DECIMAL, )), DataType::Decimal64(precision, scale) => Ok(DataTypeMap::new( DataType::Decimal64(*precision, *scale), PythonType::Float, SqlType::DECIMAL, )), DataType::Decimal128(precision, scale) => Ok(DataTypeMap::new( DataType::Decimal128(*precision, *scale), PythonType::Float, SqlType::DECIMAL, )), DataType::Decimal256(precision, scale) => Ok(DataTypeMap::new( DataType::Decimal256(*precision, *scale), PythonType::Float, SqlType::DECIMAL, )), DataType::Map(_, _) => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::RunEndEncoded(_, _) => { Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))) } DataType::BinaryView => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::Utf8View => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::ListView(_) => Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))), DataType::LargeListView(_) => { Err(PyNotImplementedError::new_err(format!("{arrow_type:?}"))) } } } /// Generate the `DataTypeMap` from a `ScalarValue` instance pub fn map_from_scalar_value(scalar_val: &ScalarValue) -> Result { DataTypeMap::map_from_arrow_type(&DataTypeMap::map_from_scalar_to_arrow(scalar_val)?) } /// Maps a `ScalarValue` to an Arrow `DataType` pub fn map_from_scalar_to_arrow(scalar_val: &ScalarValue) -> Result { match scalar_val { ScalarValue::Boolean(_) => Ok(DataType::Boolean), ScalarValue::Float16(_) => Ok(DataType::Float16), ScalarValue::Float32(_) => Ok(DataType::Float32), ScalarValue::Float64(_) => Ok(DataType::Float64), ScalarValue::Decimal32(_, precision, scale) => { Ok(DataType::Decimal32(*precision, *scale)) } ScalarValue::Decimal64(_, precision, scale) => { Ok(DataType::Decimal64(*precision, *scale)) } ScalarValue::Decimal128(_, precision, scale) => { Ok(DataType::Decimal128(*precision, *scale)) } ScalarValue::Decimal256(_, precision, scale) => { Ok(DataType::Decimal256(*precision, *scale)) } ScalarValue::Dictionary(data_type, scalar_type) => { // Call this function again to map the dictionary scalar_value to an Arrow type Ok(DataType::Dictionary( Box::new(*data_type.clone()), Box::new(DataTypeMap::map_from_scalar_to_arrow(scalar_type)?), )) } ScalarValue::Int8(_) => Ok(DataType::Int8), ScalarValue::Int16(_) => Ok(DataType::Int16), ScalarValue::Int32(_) => Ok(DataType::Int32), ScalarValue::Int64(_) => Ok(DataType::Int64), ScalarValue::UInt8(_) => Ok(DataType::UInt8), ScalarValue::UInt16(_) => Ok(DataType::UInt16), ScalarValue::UInt32(_) => Ok(DataType::UInt32), ScalarValue::UInt64(_) => Ok(DataType::UInt64), ScalarValue::Utf8(_) => Ok(DataType::Utf8), ScalarValue::LargeUtf8(_) => Ok(DataType::LargeUtf8), ScalarValue::Binary(_) => Ok(DataType::Binary), ScalarValue::LargeBinary(_) => Ok(DataType::LargeBinary), ScalarValue::Date32(_) => Ok(DataType::Date32), ScalarValue::Date64(_) => Ok(DataType::Date64), ScalarValue::Time32Second(_) => Ok(DataType::Time32(TimeUnit::Second)), ScalarValue::Time32Millisecond(_) => Ok(DataType::Time32(TimeUnit::Millisecond)), ScalarValue::Time64Microsecond(_) => Ok(DataType::Time64(TimeUnit::Microsecond)), ScalarValue::Time64Nanosecond(_) => Ok(DataType::Time64(TimeUnit::Nanosecond)), ScalarValue::Null => Ok(DataType::Null), ScalarValue::TimestampSecond(_, tz) => { Ok(DataType::Timestamp(TimeUnit::Second, tz.to_owned())) } ScalarValue::TimestampMillisecond(_, tz) => { Ok(DataType::Timestamp(TimeUnit::Millisecond, tz.to_owned())) } ScalarValue::TimestampMicrosecond(_, tz) => { Ok(DataType::Timestamp(TimeUnit::Microsecond, tz.to_owned())) } ScalarValue::TimestampNanosecond(_, tz) => { Ok(DataType::Timestamp(TimeUnit::Nanosecond, tz.to_owned())) } ScalarValue::IntervalYearMonth(..) => Ok(DataType::Interval(IntervalUnit::YearMonth)), ScalarValue::IntervalDayTime(..) => Ok(DataType::Interval(IntervalUnit::DayTime)), ScalarValue::IntervalMonthDayNano(..) => { Ok(DataType::Interval(IntervalUnit::MonthDayNano)) } ScalarValue::List(arr) => Ok(arr.data_type().to_owned()), ScalarValue::LargeList(arr) => Ok(arr.data_type().to_owned()), ScalarValue::ListView(arr) => Ok(arr.data_type().to_owned()), ScalarValue::LargeListView(arr) => Ok(arr.data_type().to_owned()), ScalarValue::Struct(_fields) => Err(PyNotImplementedError::new_err( "ScalarValue::Struct".to_string(), )), ScalarValue::FixedSizeBinary(size, _) => Ok(DataType::FixedSizeBinary(*size)), ScalarValue::FixedSizeList(_array_ref) => { // The FieldRef was removed from ScalarValue::FixedSizeList in // https://github.com/apache/arrow-datafusion/pull/8221, so we can no // longer convert back to a DataType here Err(PyNotImplementedError::new_err( "ScalarValue::FixedSizeList".to_string(), )) } ScalarValue::DurationSecond(_) => Ok(DataType::Duration(TimeUnit::Second)), ScalarValue::DurationMillisecond(_) => Ok(DataType::Duration(TimeUnit::Millisecond)), ScalarValue::DurationMicrosecond(_) => Ok(DataType::Duration(TimeUnit::Microsecond)), ScalarValue::DurationNanosecond(_) => Ok(DataType::Duration(TimeUnit::Nanosecond)), ScalarValue::Union(_, _, _) => Err(PyNotImplementedError::new_err( "ScalarValue::LargeList".to_string(), )), ScalarValue::Utf8View(_) => Ok(DataType::Utf8View), ScalarValue::BinaryView(_) => Ok(DataType::BinaryView), ScalarValue::Map(_) => Err(PyNotImplementedError::new_err( "ScalarValue::Map".to_string(), )), ScalarValue::RunEndEncoded(field1, field2, _) => Ok(DataType::RunEndEncoded( Arc::clone(field1), Arc::clone(field2), )), } } } #[pymethods] impl DataTypeMap { #[new] pub fn py_new(arrow_type: PyDataType, python_type: PythonType, sql_type: SqlType) -> Self { DataTypeMap { arrow_type, python_type, sql_type, } } #[staticmethod] #[pyo3(name = "from_parquet_type_str")] /// When using pyarrow.parquet.read_metadata().schema.column(x).physical_type you are presented /// with a String type for schema rather than an object type. Here we make a best effort /// to convert that to a physical type. pub fn py_map_from_parquet_type_str(parquet_str_type: String) -> PyResult { let arrow_dtype = match parquet_str_type.to_lowercase().as_str() { "boolean" => Ok(DataType::Boolean), "int32" => Ok(DataType::Int32), "int64" => Ok(DataType::Int64), "int96" => { // Int96 is an old parquet datatype that is now deprecated. We convert to nanosecond timestamp Ok(DataType::Timestamp(TimeUnit::Nanosecond, None)) } "float" => Ok(DataType::Float32), "double" => Ok(DataType::Float64), "byte_array" => Ok(DataType::Utf8), _ => Err(PyValueError::new_err(format!( "Unable to determine Arrow Data Type from Parquet String type: {parquet_str_type:?}" ))), }; DataTypeMap::map_from_arrow_type(&arrow_dtype?) } #[staticmethod] #[pyo3(name = "arrow")] pub fn py_map_from_arrow_type(arrow_type: &PyDataType) -> PyResult { DataTypeMap::map_from_arrow_type(&arrow_type.data_type) } #[staticmethod] #[pyo3(name = "arrow_str")] pub fn py_map_from_arrow_type_str(arrow_type_str: String) -> PyResult { let data_type = PyDataType::py_map_from_arrow_type_str(arrow_type_str); DataTypeMap::map_from_arrow_type(&data_type?.data_type) } #[staticmethod] #[pyo3(name = "sql")] pub fn py_map_from_sql_type(sql_type: &SqlType) -> PyResult { match sql_type { SqlType::ANY => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::ARRAY => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::BIGINT => Ok(DataTypeMap::new( DataType::Int64, PythonType::Int, SqlType::BIGINT, )), SqlType::BINARY => Ok(DataTypeMap::new( DataType::Binary, PythonType::Bytes, SqlType::BINARY, )), SqlType::BOOLEAN => Ok(DataTypeMap::new( DataType::Boolean, PythonType::Bool, SqlType::BOOLEAN, )), SqlType::CHAR => Ok(DataTypeMap::new( DataType::UInt8, PythonType::Int, SqlType::CHAR, )), SqlType::COLUMN_LIST => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::CURSOR => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::DATE => Ok(DataTypeMap::new( DataType::Date64, PythonType::Datetime, SqlType::DATE, )), SqlType::DECIMAL => Ok(DataTypeMap::new( DataType::Decimal128(1, 1), PythonType::Float, SqlType::DECIMAL, )), SqlType::DISTINCT => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::DOUBLE => Ok(DataTypeMap::new( DataType::Decimal256(1, 1), PythonType::Float, SqlType::DOUBLE, )), SqlType::DYNAMIC_STAR => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::FLOAT => Ok(DataTypeMap::new( DataType::Decimal128(1, 1), PythonType::Float, SqlType::FLOAT, )), SqlType::GEOMETRY => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::INTEGER => Ok(DataTypeMap::new( DataType::Int8, PythonType::Int, SqlType::INTEGER, )), SqlType::INTERVAL => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::INTERVAL_DAY => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::INTERVAL_DAY_HOUR => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_DAY_MINUTE => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_DAY_SECOND => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_HOUR => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::INTERVAL_HOUR_MINUTE => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_HOUR_SECOND => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_MINUTE => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_MINUTE_SECOND => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_MONTH => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::INTERVAL_SECOND => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::INTERVAL_YEAR => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::INTERVAL_YEAR_MONTH => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::MAP => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::MULTISET => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::NULL => Ok(DataTypeMap::new( DataType::Null, PythonType::None, SqlType::NULL, )), SqlType::OTHER => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::REAL => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::ROW => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::SARG => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::SMALLINT => Ok(DataTypeMap::new( DataType::Int16, PythonType::Int, SqlType::SMALLINT, )), SqlType::STRUCTURED => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::SYMBOL => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::TIME => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::TIME_WITH_LOCAL_TIME_ZONE => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::TIMESTAMP => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::TIMESTAMP_WITH_LOCAL_TIME_ZONE => { Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))) } SqlType::TINYINT => Ok(DataTypeMap::new( DataType::Int8, PythonType::Int, SqlType::TINYINT, )), SqlType::UNKNOWN => Err(PyNotImplementedError::new_err(format!("{sql_type:?}"))), SqlType::VARBINARY => Ok(DataTypeMap::new( DataType::LargeBinary, PythonType::Bytes, SqlType::VARBINARY, )), SqlType::VARCHAR => Ok(DataTypeMap::new( DataType::Utf8, PythonType::Str, SqlType::VARCHAR, )), } } /// Unfortunately PyO3 does not allow for us to expose the DataType as an enum since /// we cannot directly annotate the Enum instance of dependency code. Therefore, here /// we provide an enum to mimic it. #[pyo3(name = "friendly_arrow_type_name")] pub fn friendly_arrow_type_name(&self) -> PyResult<&str> { Ok(match &self.arrow_type.data_type { DataType::Null => "Null", DataType::Boolean => "Boolean", DataType::Int8 => "Int8", DataType::Int16 => "Int16", DataType::Int32 => "Int32", DataType::Int64 => "Int64", DataType::UInt8 => "UInt8", DataType::UInt16 => "UInt16", DataType::UInt32 => "UInt32", DataType::UInt64 => "UInt64", DataType::Float16 => "Float16", DataType::Float32 => "Float32", DataType::Float64 => "Float64", DataType::Timestamp(_, _) => "Timestamp", DataType::Date32 => "Date32", DataType::Date64 => "Date64", DataType::Time32(_) => "Time32", DataType::Time64(_) => "Time64", DataType::Duration(_) => "Duration", DataType::Interval(_) => "Interval", DataType::Binary => "Binary", DataType::FixedSizeBinary(_) => "FixedSizeBinary", DataType::LargeBinary => "LargeBinary", DataType::Utf8 => "Utf8", DataType::LargeUtf8 => "LargeUtf8", DataType::List(_) => "List", DataType::FixedSizeList(_, _) => "FixedSizeList", DataType::LargeList(_) => "LargeList", DataType::Struct(_) => "Struct", DataType::Union(_, _) => "Union", DataType::Dictionary(_, _) => "Dictionary", DataType::Decimal32(_, _) => "Decimal32", DataType::Decimal64(_, _) => "Decimal64", DataType::Decimal128(_, _) => "Decimal128", DataType::Decimal256(_, _) => "Decimal256", DataType::Map(_, _) => "Map", DataType::RunEndEncoded(_, _) => "RunEndEncoded", DataType::BinaryView => "BinaryView", DataType::Utf8View => "Utf8View", DataType::ListView(_) => "ListView", DataType::LargeListView(_) => "LargeListView", }) } } /// PyO3 requires that objects passed between Rust and Python implement the trait `PyClass` /// Since `DataType` exists in another package we cannot make that happen here so we wrap /// `DataType` as `PyDataType` This exists solely to satisfy those constraints. #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, name = "DataType", module = "datafusion.common" )] pub struct PyDataType { pub data_type: DataType, } impl PyDataType { /// There are situations when obtaining dtypes on the Python side where the Arrow type /// is presented as a String rather than an actual DataType. This function is used to /// convert that String to a DataType for the Python side to use. pub fn py_map_from_arrow_type_str(arrow_str_type: String) -> PyResult { // Certain string types contain "metadata" that should be trimmed here. Ex: "datetime64[ns, Europe/Berlin]" let arrow_str_type = match arrow_str_type.find('[') { Some(index) => arrow_str_type[0..index].to_string(), None => arrow_str_type, // Return early if ',' is not found. }; let arrow_dtype = match arrow_str_type.to_lowercase().as_str() { "bool" => Ok(DataType::Boolean), "boolean" => Ok(DataType::Boolean), "uint8" => Ok(DataType::UInt8), "uint16" => Ok(DataType::UInt16), "uint32" => Ok(DataType::UInt32), "uint64" => Ok(DataType::UInt64), "int8" => Ok(DataType::Int8), "int16" => Ok(DataType::Int16), "int32" => Ok(DataType::Int32), "int64" => Ok(DataType::Int64), "float" => Ok(DataType::Float32), "double" => Ok(DataType::Float64), "float16" => Ok(DataType::Float16), "float32" => Ok(DataType::Float32), "float64" => Ok(DataType::Float64), "datetime64" => Ok(DataType::Date64), "object" => Ok(DataType::Utf8), _ => Err(PyValueError::new_err(format!( "Unable to determine Arrow Data Type from Arrow String type: {arrow_str_type:?}" ))), }; Ok(PyDataType { data_type: arrow_dtype?, }) } } impl From for DataType { fn from(data_type: PyDataType) -> DataType { data_type.data_type } } impl From for PyDataType { fn from(data_type: DataType) -> PyDataType { PyDataType { data_type } } } /// Represents the possible Python types that can be mapped to the SQL types #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "PythonType", module = "datafusion.common" )] pub enum PythonType { Array, Bool, Bytes, Datetime, Float, Int, List, None, Object, Str, } /// Represents the types that are possible for DataFusion to parse /// from a SQL query. Aka "SqlType" and are valid values for /// ANSI SQL #[allow(non_camel_case_types)] #[allow(clippy::upper_case_acronyms)] #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "SqlType", module = "datafusion.common" )] pub enum SqlType { ANY, ARRAY, BIGINT, BINARY, BOOLEAN, CHAR, COLUMN_LIST, CURSOR, DATE, DECIMAL, DISTINCT, DOUBLE, DYNAMIC_STAR, FLOAT, GEOMETRY, INTEGER, INTERVAL, INTERVAL_DAY, INTERVAL_DAY_HOUR, INTERVAL_DAY_MINUTE, INTERVAL_DAY_SECOND, INTERVAL_HOUR, INTERVAL_HOUR_MINUTE, INTERVAL_HOUR_SECOND, INTERVAL_MINUTE, INTERVAL_MINUTE_SECOND, INTERVAL_MONTH, INTERVAL_SECOND, INTERVAL_YEAR, INTERVAL_YEAR_MONTH, MAP, MULTISET, NULL, OTHER, REAL, ROW, SARG, SMALLINT, STRUCTURED, SYMBOL, TIME, TIME_WITH_LOCAL_TIME_ZONE, TIMESTAMP, TIMESTAMP_WITH_LOCAL_TIME_ZONE, TINYINT, UNKNOWN, VARBINARY, VARCHAR, } /// Specifies Ignore / Respect NULL within window functions. /// For example /// `FIRST_VALUE(column2) IGNORE NULLS OVER (PARTITION BY column1)` #[allow(non_camel_case_types)] #[allow(clippy::upper_case_acronyms)] #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "NullTreatment", module = "datafusion.common" )] pub enum NullTreatment { IGNORE_NULLS, RESPECT_NULLS, } impl From for DFNullTreatment { fn from(null_treatment: NullTreatment) -> DFNullTreatment { match null_treatment { NullTreatment::IGNORE_NULLS => DFNullTreatment::IgnoreNulls, NullTreatment::RESPECT_NULLS => DFNullTreatment::RespectNulls, } } } impl From for NullTreatment { fn from(null_treatment: DFNullTreatment) -> NullTreatment { match null_treatment { DFNullTreatment::IgnoreNulls => NullTreatment::IGNORE_NULLS, DFNullTreatment::RespectNulls => NullTreatment::RESPECT_NULLS, } } } apache-datafusion-python-54.0.0/crates/core/src/common/df_schema.rs000066400000000000000000000032421521727353300252530ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::common::DFSchema; use pyo3::prelude::*; #[derive(Debug, Clone)] #[pyclass( from_py_object, frozen, name = "DFSchema", module = "datafusion.common", subclass )] pub struct PyDFSchema { schema: Arc, } impl From for DFSchema { fn from(schema: PyDFSchema) -> DFSchema { (*schema.schema).clone() } } impl From for PyDFSchema { fn from(schema: DFSchema) -> PyDFSchema { PyDFSchema { schema: Arc::new(schema), } } } #[pymethods] impl PyDFSchema { #[pyo3(name = "empty")] #[staticmethod] fn py_empty() -> PyResult { Ok(Self { schema: Arc::new(DFSchema::empty()), }) } #[pyo3(name = "field_names")] fn py_field_names(&self) -> PyResult> { Ok(self.schema.field_names()) } } apache-datafusion-python-54.0.0/crates/core/src/common/function.rs000066400000000000000000000035321521727353300251710ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use datafusion::arrow::datatypes::DataType; use pyo3::prelude::*; use super::data_type::PyDataType; #[pyclass( from_py_object, frozen, name = "SqlFunction", module = "datafusion.common", subclass )] #[derive(Debug, Clone)] pub struct SqlFunction { pub name: String, pub return_types: HashMap, DataType>, pub aggregation: bool, } impl SqlFunction { pub fn new( function_name: String, input_types: Vec, return_type: PyDataType, aggregation_bool: bool, ) -> Self { let mut func = Self { name: function_name, return_types: HashMap::new(), aggregation: aggregation_bool, }; func.add_type_mapping(input_types, return_type); func } pub fn add_type_mapping(&mut self, input_types: Vec, return_type: PyDataType) { self.return_types.insert( input_types.iter().map(|t| t.clone().into()).collect(), return_type.into(), ); } } apache-datafusion-python-54.0.0/crates/core/src/common/schema.rs000066400000000000000000000237401521727353300246070ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::borrow::Cow; use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use arrow::datatypes::Schema; use arrow::pyarrow::PyArrowType; use datafusion::arrow::datatypes::SchemaRef; use datafusion::common::Constraints; use datafusion::datasource::TableType; use datafusion::logical_expr::utils::split_conjunction; use datafusion::logical_expr::{Expr, TableProviderFilterPushDown, TableSource}; use parking_lot::RwLock; use pyo3::prelude::*; use super::data_type::DataTypeMap; use super::function::SqlFunction; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, name = "SqlSchema", module = "datafusion.common", subclass, frozen )] #[derive(Debug, Clone)] pub struct SqlSchema { name: Arc>, tables: Arc>>, views: Arc>>, functions: Arc>>, } #[pyclass( from_py_object, name = "SqlTable", module = "datafusion.common", subclass )] #[derive(Debug, Clone)] pub struct SqlTable { #[pyo3(get, set)] pub name: String, #[pyo3(get, set)] pub columns: Vec<(String, DataTypeMap)>, #[pyo3(get, set)] pub primary_key: Option, #[pyo3(get, set)] pub foreign_keys: Vec, #[pyo3(get, set)] pub indexes: Vec, #[pyo3(get, set)] pub constraints: Vec, #[pyo3(get, set)] pub statistics: SqlStatistics, #[pyo3(get, set)] pub filepaths: Option>, } #[pymethods] impl SqlTable { #[new] #[pyo3(signature = (table_name, columns, row_count, filepaths=None))] pub fn new( table_name: String, columns: Vec<(String, DataTypeMap)>, row_count: f64, filepaths: Option>, ) -> Self { Self { name: table_name, columns, primary_key: None, foreign_keys: Vec::new(), indexes: Vec::new(), constraints: Vec::new(), statistics: SqlStatistics::new(row_count), filepaths, } } } #[pyclass( from_py_object, name = "SqlView", module = "datafusion.common", subclass )] #[derive(Debug, Clone)] pub struct SqlView { #[pyo3(get, set)] pub name: String, #[pyo3(get, set)] pub definition: String, // SQL code that defines the view } #[pymethods] impl SqlSchema { #[new] pub fn new(schema_name: &str) -> Self { Self { name: Arc::new(RwLock::new(schema_name.to_owned())), tables: Arc::new(RwLock::new(Vec::new())), views: Arc::new(RwLock::new(Vec::new())), functions: Arc::new(RwLock::new(Vec::new())), } } #[getter] fn name(&self) -> PyResult { Ok(self.name.read().clone()) } #[setter] fn set_name(&self, value: String) -> PyResult<()> { *self.name.write() = value; Ok(()) } #[getter] fn tables(&self) -> PyResult> { Ok(self.tables.read().clone()) } #[setter] fn set_tables(&self, tables: Vec) -> PyResult<()> { *self.tables.write() = tables; Ok(()) } #[getter] fn views(&self) -> PyResult> { Ok(self.views.read().clone()) } #[setter] fn set_views(&self, views: Vec) -> PyResult<()> { *self.views.write() = views; Ok(()) } #[getter] fn functions(&self) -> PyResult> { Ok(self.functions.read().clone()) } #[setter] fn set_functions(&self, functions: Vec) -> PyResult<()> { *self.functions.write() = functions; Ok(()) } pub fn table_by_name(&self, table_name: &str) -> Option { let tables = self.tables.read(); tables.iter().find(|tbl| tbl.name.eq(table_name)).cloned() } pub fn add_table(&self, table: SqlTable) { let mut tables = self.tables.write(); tables.push(table); } pub fn drop_table(&self, table_name: String) { let mut tables = self.tables.write(); tables.retain(|x| !x.name.eq(&table_name)); } } /// SqlTable wrapper that is compatible with DataFusion logical query plans pub struct SqlTableSource { schema: SchemaRef, statistics: Option, filepaths: Option>, } impl SqlTableSource { /// Initialize a new `EmptyTable` from a schema pub fn new( schema: SchemaRef, statistics: Option, filepaths: Option>, ) -> Self { Self { schema, statistics, filepaths, } } /// Access optional statistics associated with this table source pub fn statistics(&self) -> Option<&SqlStatistics> { self.statistics.as_ref() } /// Access optional filepath associated with this table source #[allow(dead_code)] pub fn filepaths(&self) -> Option<&Vec> { self.filepaths.as_ref() } } /// Implement TableSource, used in the logical query plan and in logical query optimizations impl TableSource for SqlTableSource { fn schema(&self) -> SchemaRef { self.schema.clone() } fn table_type(&self) -> datafusion::logical_expr::TableType { datafusion::logical_expr::TableType::Base } fn supports_filters_pushdown( &self, filters: &[&Expr], ) -> datafusion::common::Result> { filters .iter() .map(|f| { let filters = split_conjunction(f); if filters.iter().all(|f| is_supported_push_down_expr(f)) { // Push down filters to the tablescan operation if all are supported Ok(TableProviderFilterPushDown::Exact) } else if filters.iter().any(|f| is_supported_push_down_expr(f)) { // Partially apply the filter in the TableScan but retain // the Filter operator in the plan as well Ok(TableProviderFilterPushDown::Inexact) } else { Ok(TableProviderFilterPushDown::Unsupported) } }) .collect() } fn get_logical_plan(&self) -> Option> { None } } fn is_supported_push_down_expr(_expr: &Expr) -> bool { // For now we support all kinds of expr's at this level true } #[pyclass( from_py_object, frozen, name = "SqlStatistics", module = "datafusion.common", subclass )] #[derive(Debug, Clone)] pub struct SqlStatistics { row_count: f64, } #[pymethods] impl SqlStatistics { #[new] pub fn new(row_count: f64) -> Self { Self { row_count } } #[pyo3(name = "getRowCount")] pub fn get_row_count(&self) -> f64 { self.row_count } } #[pyclass( from_py_object, frozen, name = "Constraints", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyConstraints { pub constraints: Constraints, } impl From for Constraints { fn from(constraints: PyConstraints) -> Self { constraints.constraints } } impl From for PyConstraints { fn from(constraints: Constraints) -> Self { PyConstraints { constraints } } } impl Display for PyConstraints { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "Constraints: {:?}", self.constraints) } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "TableType", module = "datafusion.common" )] pub enum PyTableType { Base, View, Temporary, } impl From for datafusion::logical_expr::TableType { fn from(table_type: PyTableType) -> Self { match table_type { PyTableType::Base => datafusion::logical_expr::TableType::Base, PyTableType::View => datafusion::logical_expr::TableType::View, PyTableType::Temporary => datafusion::logical_expr::TableType::Temporary, } } } impl From for PyTableType { fn from(table_type: TableType) -> Self { match table_type { datafusion::logical_expr::TableType::Base => PyTableType::Base, datafusion::logical_expr::TableType::View => PyTableType::View, datafusion::logical_expr::TableType::Temporary => PyTableType::Temporary, } } } #[pyclass( from_py_object, frozen, name = "TableSource", module = "datafusion.common", subclass )] #[derive(Clone)] pub struct PyTableSource { pub table_source: Arc, } #[pymethods] impl PyTableSource { pub fn schema(&self) -> PyArrowType { (*self.table_source.schema()).clone().into() } pub fn constraints(&self) -> Option { self.table_source.constraints().map(|c| PyConstraints { constraints: c.clone(), }) } pub fn table_type(&self) -> PyTableType { self.table_source.table_type().into() } pub fn get_logical_plan(&self) -> Option { self.table_source .get_logical_plan() .map(|plan| PyLogicalPlan::new(plan.into_owned())) } } apache-datafusion-python-54.0.0/crates/core/src/context.rs000066400000000000000000001577541521727353300235600ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::{HashMap, HashSet}; use std::path::{Path, PathBuf}; use std::ptr::NonNull; use std::str::FromStr; use std::sync::Arc; use arrow::array::RecordBatchReader; use arrow::ffi_stream::ArrowArrayStreamReader; use arrow::pyarrow::FromPyArrow; use datafusion::arrow::datatypes::{DataType, Schema, SchemaRef}; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::arrow::record_batch::RecordBatch; use datafusion::catalog::{CatalogProvider, CatalogProviderList, TableProviderFactory}; use datafusion::common::{DFSchema, ScalarValue, TableReference, exec_err}; use datafusion::datasource::file_format::file_compression_type::FileCompressionType; use datafusion::datasource::file_format::parquet::ParquetFormat; use datafusion::datasource::listing::{ ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl, }; use datafusion::datasource::{MemTable, TableProvider}; use datafusion::execution::context::{ DataFilePaths, SQLOptions, SessionConfig, SessionContext, TaskContext, }; use datafusion::execution::disk_manager::DiskManagerMode; use datafusion::execution::memory_pool::{FairSpillPool, GreedyMemoryPool, UnboundedMemoryPool}; use datafusion::execution::options::{ArrowReadOptions, ReadOptions}; use datafusion::execution::runtime_env::RuntimeEnvBuilder; use datafusion::execution::session_state::SessionStateBuilder; use datafusion::execution::{FunctionRegistry, TaskContextProvider}; use datafusion::prelude::{ AvroReadOptions, CsvReadOptions, DataFrame, JsonReadOptions, ParquetReadOptions, }; use datafusion_ffi::catalog_provider::FFI_CatalogProvider; use datafusion_ffi::catalog_provider_list::FFI_CatalogProviderList; use datafusion_ffi::config::extension_options::FFI_ExtensionOptions; use datafusion_ffi::execution::FFI_TaskContextProvider; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec; use datafusion_ffi::table_provider_factory::FFI_TableProviderFactory; use datafusion_proto::logical_plan::LogicalExtensionCodec; use datafusion_proto::physical_plan::PhysicalExtensionCodec; use datafusion_python_util::{ create_logical_extension_capsule, create_physical_extension_capsule, ffi_logical_codec_from_pycapsule, get_global_ctx, get_tokio_runtime, physical_codec_from_pycapsule, physical_optimizer_rule_from_pycapsule, spawn_future, wait_for_future, }; use object_store::ObjectStore; use pyo3::IntoPyObjectExt; use pyo3::exceptions::{PyKeyError, PyRuntimeError, PyValueError}; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyDict, PyList, PyTuple}; use url::Url; use uuid::Uuid; use crate::catalog::{ PyCatalog, PyCatalogList, RustWrappedPyCatalogProvider, RustWrappedPyCatalogProviderList, }; use crate::codec::{PythonLogicalCodec, PythonPhysicalCodec}; use crate::common::data_type::PyScalarValue; use crate::common::df_schema::PyDFSchema; use crate::dataframe::PyDataFrame; use crate::dataset::Dataset; use crate::errors::{ PyDataFusionError, PyDataFusionResult, from_datafusion_error, py_datafusion_err, }; use crate::expr::PyExpr; use crate::expr::sort_expr::PySortExpr; use crate::options::PyCsvReadOptions; use crate::physical_plan::PyExecutionPlan; use crate::record_batch::PyRecordBatchStream; use crate::sql::logical::PyLogicalPlan; use crate::sql::util::replace_placeholders_with_strings; use crate::store::StorageContexts; use crate::table::{PyTable, RustWrappedPyTableProviderFactory}; use crate::udaf::PyAggregateUDF; use crate::udf::PyScalarUDF; use crate::udtf::PyTableFunction; use crate::udwf::PyWindowUDF; /// Configuration options for a SessionContext #[pyclass( from_py_object, frozen, name = "SessionConfig", module = "datafusion", subclass )] #[derive(Clone, Default)] pub struct PySessionConfig { pub config: SessionConfig, } impl From for PySessionConfig { fn from(config: SessionConfig) -> Self { Self { config } } } #[pymethods] impl PySessionConfig { #[pyo3(signature = (config_options=None))] #[new] fn new(config_options: Option>) -> Self { let mut config = SessionConfig::new(); if let Some(hash_map) = config_options { for (k, v) in &hash_map { config = config.set(k, &ScalarValue::Utf8(Some(v.clone()))); } } Self { config } } fn with_create_default_catalog_and_schema(&self, enabled: bool) -> Self { Self::from( self.config .clone() .with_create_default_catalog_and_schema(enabled), ) } fn with_default_catalog_and_schema(&self, catalog: &str, schema: &str) -> Self { Self::from( self.config .clone() .with_default_catalog_and_schema(catalog, schema), ) } fn with_information_schema(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_information_schema(enabled)) } fn with_batch_size(&self, batch_size: usize) -> Self { Self::from(self.config.clone().with_batch_size(batch_size)) } fn with_target_partitions(&self, target_partitions: usize) -> Self { Self::from( self.config .clone() .with_target_partitions(target_partitions), ) } fn with_repartition_aggregations(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_repartition_aggregations(enabled)) } fn with_repartition_joins(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_repartition_joins(enabled)) } fn with_repartition_windows(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_repartition_windows(enabled)) } fn with_repartition_sorts(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_repartition_sorts(enabled)) } fn with_repartition_file_scans(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_repartition_file_scans(enabled)) } fn with_repartition_file_min_size(&self, size: usize) -> Self { Self::from(self.config.clone().with_repartition_file_min_size(size)) } fn with_parquet_pruning(&self, enabled: bool) -> Self { Self::from(self.config.clone().with_parquet_pruning(enabled)) } fn set(&self, key: &str, value: &str) -> Self { Self::from(self.config.clone().set_str(key, value)) } pub fn with_extension(&self, extension: Bound) -> PyResult { if !extension.hasattr("__datafusion_extension_options__")? { return Err(pyo3::exceptions::PyAttributeError::new_err( "Expected extension object to define __datafusion_extension_options__()", )); } let capsule = extension.call_method0("__datafusion_extension_options__")?; let capsule = capsule.cast::()?; let extension: NonNull = capsule .pointer_checked(Some(c"datafusion_extension_options"))? .cast(); let mut extension = unsafe { extension.as_ref() }.clone(); let mut config = self.config.clone(); let options = config.options_mut(); if let Some(prior_extension) = options.extensions.get::() { extension .merge(prior_extension) .map_err(py_datafusion_err)?; } options.extensions.insert(extension); Ok(Self::from(config)) } } /// Runtime options for a SessionContext #[pyclass( from_py_object, frozen, name = "RuntimeEnvBuilder", module = "datafusion", subclass )] #[derive(Clone)] pub struct PyRuntimeEnvBuilder { pub builder: RuntimeEnvBuilder, } #[pymethods] impl PyRuntimeEnvBuilder { #[new] fn new() -> Self { Self { builder: RuntimeEnvBuilder::default(), } } fn with_disk_manager_disabled(&self) -> Self { let mut runtime_builder = self.builder.clone(); let mut disk_mgr_builder = runtime_builder .disk_manager_builder .clone() .unwrap_or_default(); disk_mgr_builder.set_mode(DiskManagerMode::Disabled); runtime_builder = runtime_builder.with_disk_manager_builder(disk_mgr_builder); Self { builder: runtime_builder, } } fn with_disk_manager_os(&self) -> Self { let mut runtime_builder = self.builder.clone(); let mut disk_mgr_builder = runtime_builder .disk_manager_builder .clone() .unwrap_or_default(); disk_mgr_builder.set_mode(DiskManagerMode::OsTmpDirectory); runtime_builder = runtime_builder.with_disk_manager_builder(disk_mgr_builder); Self { builder: runtime_builder, } } fn with_disk_manager_specified(&self, paths: Vec) -> Self { let paths = paths.iter().map(|s| s.into()).collect(); let mut runtime_builder = self.builder.clone(); let mut disk_mgr_builder = runtime_builder .disk_manager_builder .clone() .unwrap_or_default(); disk_mgr_builder.set_mode(DiskManagerMode::Directories(paths)); runtime_builder = runtime_builder.with_disk_manager_builder(disk_mgr_builder); Self { builder: runtime_builder, } } fn with_unbounded_memory_pool(&self) -> Self { let builder = self.builder.clone(); let builder = builder.with_memory_pool(Arc::new(UnboundedMemoryPool::default())); Self { builder } } fn with_fair_spill_pool(&self, size: usize) -> Self { let builder = self.builder.clone(); let builder = builder.with_memory_pool(Arc::new(FairSpillPool::new(size))); Self { builder } } fn with_greedy_memory_pool(&self, size: usize) -> Self { let builder = self.builder.clone(); let builder = builder.with_memory_pool(Arc::new(GreedyMemoryPool::new(size))); Self { builder } } fn with_temp_file_path(&self, path: &str) -> Self { let builder = self.builder.clone(); let builder = builder.with_temp_file_path(path); Self { builder } } } /// `PySQLOptions` allows you to specify options to the sql execution. #[pyclass( from_py_object, frozen, name = "SQLOptions", module = "datafusion", subclass )] #[derive(Clone)] pub struct PySQLOptions { pub options: SQLOptions, } impl From for PySQLOptions { fn from(options: SQLOptions) -> Self { Self { options } } } #[pymethods] impl PySQLOptions { #[new] fn new() -> Self { let options = SQLOptions::new(); Self { options } } /// Should DDL data modification commands (e.g. `CREATE TABLE`) be run? Defaults to `true`. fn with_allow_ddl(&self, allow: bool) -> Self { Self::from(self.options.with_allow_ddl(allow)) } /// Should DML data modification commands (e.g. `INSERT and COPY`) be run? Defaults to `true` pub fn with_allow_dml(&self, allow: bool) -> Self { Self::from(self.options.with_allow_dml(allow)) } /// Should Statements such as (e.g. `SET VARIABLE and `BEGIN TRANSACTION` ...`) be run?. Defaults to `true` pub fn with_allow_statements(&self, allow: bool) -> Self { Self::from(self.options.with_allow_statements(allow)) } } /// `PySessionContext` is able to plan and execute DataFusion plans. /// It has a powerful optimizer, a physical planner for local execution, and a /// multi-threaded execution engine to perform the execution. #[pyclass( from_py_object, frozen, name = "SessionContext", module = "datafusion", subclass )] #[derive(Clone)] pub struct PySessionContext { pub ctx: Arc, logical_codec: Arc, physical_codec: Arc, } #[pymethods] impl PySessionContext { #[pyo3(signature = (config=None, runtime=None))] #[new] pub fn new( config: Option, runtime: Option, ) -> PyDataFusionResult { let config = if let Some(c) = config { c.config } else { SessionConfig::default().with_information_schema(true) }; let runtime_env_builder = if let Some(c) = runtime { c.builder } else { RuntimeEnvBuilder::default() }; let runtime = Arc::new(runtime_env_builder.build()?); let session_state = SessionStateBuilder::new() .with_config(config) .with_runtime_env(runtime) .with_default_features() .with_analyzer_rule(Arc::new(crate::analyzer::ResolveLambdaVariables::new())) .build(); let ctx = Arc::new(SessionContext::new_with_state(session_state)); Ok(PySessionContext { ctx, logical_codec: Arc::new(PythonLogicalCodec::default()), physical_codec: Arc::new(PythonPhysicalCodec::default()), }) } pub fn enable_url_table(&self) -> PyResult { Ok(PySessionContext { ctx: Arc::new(self.ctx.as_ref().clone().enable_url_table()), logical_codec: Arc::clone(&self.logical_codec), physical_codec: Arc::clone(&self.physical_codec), }) } #[staticmethod] #[pyo3(signature = ())] pub fn global_ctx() -> PyResult { let ctx = get_global_ctx().clone(); Ok(Self { ctx, logical_codec: Arc::new(PythonLogicalCodec::default()), physical_codec: Arc::new(PythonPhysicalCodec::default()), }) } /// Register an object store with the given name #[pyo3(signature = (scheme, store, host=None))] pub fn register_object_store( &self, scheme: &str, store: StorageContexts, host: Option<&str>, ) -> PyResult<()> { // for most stores the "host" is the bucket name and can be inferred from the store let (store, upstream_host): (Arc, String) = match store { StorageContexts::AmazonS3(s3) => (s3.inner, s3.bucket_name), StorageContexts::GoogleCloudStorage(gcs) => (gcs.inner, gcs.bucket_name), StorageContexts::MicrosoftAzure(azure) => (azure.inner, azure.container_name), StorageContexts::LocalFileSystem(local) => (local.inner, "".to_string()), StorageContexts::HTTP(http) => (http.store, http.url), }; // let users override the host to match the api signature from upstream let derived_host = if let Some(host) = host { host } else { &upstream_host }; let url_string = format!("{scheme}{derived_host}"); let url = Url::parse(&url_string).map_err(|e| PyValueError::new_err(e.to_string()))?; self.ctx.runtime_env().register_object_store(&url, store); Ok(()) } /// Deregister an object store with the given url #[pyo3(signature = (scheme, host=None))] pub fn deregister_object_store( &self, scheme: &str, host: Option<&str>, ) -> PyDataFusionResult<()> { let host = host.unwrap_or(""); let url_string = format!("{scheme}{host}"); let url = Url::parse(&url_string).map_err(|e| PyDataFusionError::Common(e.to_string()))?; self.ctx.runtime_env().deregister_object_store(&url)?; Ok(()) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = (name, path, table_partition_cols=vec![], file_extension=".parquet", schema=None, file_sort_order=None))] pub fn register_listing_table( &self, name: &str, path: PathBuf, table_partition_cols: Vec<(String, PyArrowType)>, file_extension: &str, schema: Option>, file_sort_order: Option>>, py: Python, ) -> PyDataFusionResult<()> { let options = ListingOptions::new(Arc::new(ParquetFormat::new())) .with_file_extension(file_extension) .with_table_partition_cols(convert_partition_cols(table_partition_cols)) .with_file_sort_order(convert_file_sort_order(file_sort_order)); let table_path = ListingTableUrl::parse(path_to_str(&path)?)?; let resolved_schema: SchemaRef = match schema { Some(s) => Arc::new(s.0), None => { let state = self.ctx.state(); let schema = options.infer_schema(&state, &table_path); wait_for_future(py, schema)?? } }; let config = ListingTableConfig::new(table_path) .with_listing_options(options) .with_schema(resolved_schema); let table = ListingTable::try_new(config)?; self.ctx.register_table(name, Arc::new(table))?; Ok(()) } pub fn register_udtf(&self, func: PyTableFunction) { let name = func.name.clone(); let func = Arc::new(func); self.ctx.register_udtf(&name, func); } pub fn deregister_udtf(&self, name: &str) { self.ctx.deregister_udtf(name); } #[pyo3(signature = (query, options=None, param_values=HashMap::default(), param_strings=HashMap::default()))] pub fn sql_with_options( &self, py: Python, mut query: String, options: Option, param_values: HashMap, param_strings: HashMap, ) -> PyDataFusionResult { let options = if let Some(options) = options { options.options } else { SQLOptions::new() }; let param_values = param_values .into_iter() .map(|(name, value)| (name, ScalarValue::from(value))) .collect::>(); let state = self.ctx.state(); let dialect = state.config().options().sql_parser.dialect.as_ref(); if !param_strings.is_empty() { query = replace_placeholders_with_strings(&query, dialect, param_strings)?; } let mut df = wait_for_future(py, async { self.ctx.sql_with_options(&query, options).await })? .map_err(from_datafusion_error)?; if !param_values.is_empty() { df = df.with_param_values(param_values)?; } Ok(PyDataFrame::new(df)) } #[pyo3(signature = (partitions, name=None, schema=None))] pub fn create_dataframe( &self, partitions: PyArrowType>>, name: Option<&str>, schema: Option>, py: Python, ) -> PyDataFusionResult { let schema = if let Some(schema) = schema { SchemaRef::from(schema.0) } else { partitions.0[0][0].schema() }; let table = MemTable::try_new(schema, partitions.0)?; // generate a random (unique) name for this table if none is provided // table name cannot start with numeric digit let table_name = match name { Some(val) => val.to_owned(), None => { "c".to_owned() + Uuid::new_v4() .simple() .encode_lower(&mut Uuid::encode_buffer()) } }; self.ctx.register_table(&*table_name, Arc::new(table))?; let table = wait_for_future(py, self._table(&table_name))??; let df = PyDataFrame::new(table); Ok(df) } /// Create a DataFrame from an existing logical plan pub fn create_dataframe_from_logical_plan(&self, plan: PyLogicalPlan) -> PyDataFrame { PyDataFrame::new(DataFrame::new(self.ctx.state(), plan.plan.as_ref().clone())) } /// Construct datafusion dataframe from Python list #[pyo3(signature = (data, name=None))] pub fn from_pylist( &self, data: Bound<'_, PyList>, name: Option<&str>, ) -> PyResult { // Acquire GIL Token let py = data.py(); // Instantiate pyarrow Table object & convert to Arrow Table let table_class = py.import("pyarrow")?.getattr("Table")?; let args = PyTuple::new(py, &[data])?; let table = table_class.call_method1("from_pylist", args)?; // Convert Arrow Table to datafusion DataFrame let df = self.from_arrow(table, name, py)?; Ok(df) } /// Construct datafusion dataframe from Python dictionary #[pyo3(signature = (data, name=None))] pub fn from_pydict( &self, data: Bound<'_, PyDict>, name: Option<&str>, ) -> PyResult { // Acquire GIL Token let py = data.py(); // Instantiate pyarrow Table object & convert to Arrow Table let table_class = py.import("pyarrow")?.getattr("Table")?; let args = PyTuple::new(py, &[data])?; let table = table_class.call_method1("from_pydict", args)?; // Convert Arrow Table to datafusion DataFrame let df = self.from_arrow(table, name, py)?; Ok(df) } /// Construct datafusion dataframe from Arrow Table #[pyo3(signature = (data, name=None))] pub fn from_arrow( &self, data: Bound<'_, PyAny>, name: Option<&str>, py: Python, ) -> PyDataFusionResult { let (schema, batches) = if let Ok(stream_reader) = ArrowArrayStreamReader::from_pyarrow_bound(&data) { // Works for any object that implements __arrow_c_stream__ in pycapsule. let schema = stream_reader.schema().as_ref().to_owned(); let batches = stream_reader .collect::, arrow::error::ArrowError>>()?; (schema, batches) } else if let Ok(array) = RecordBatch::from_pyarrow_bound(&data) { // While this says RecordBatch, it will work for any object that implements // __arrow_c_array__ and returns a StructArray. (array.schema().as_ref().to_owned(), vec![array]) } else { return Err(PyDataFusionError::Common( "Expected either a Arrow Array or Arrow Stream in from_arrow().".to_string(), )); }; // Because create_dataframe() expects a vector of vectors of record batches // here we need to wrap the vector of record batches in an additional vector let list_of_batches = PyArrowType::from(vec![batches]); self.create_dataframe(list_of_batches, name, Some(schema.into()), py) } /// Construct datafusion dataframe from pandas #[allow(clippy::wrong_self_convention)] #[pyo3(signature = (data, name=None))] pub fn from_pandas(&self, data: Bound<'_, PyAny>, name: Option<&str>) -> PyResult { // Obtain GIL token let py = data.py(); // Instantiate pyarrow Table object & convert to Arrow Table let table_class = py.import("pyarrow")?.getattr("Table")?; let args = PyTuple::new(py, &[data])?; let table = table_class.call_method1("from_pandas", args)?; // Convert Arrow Table to datafusion DataFrame let df = self.from_arrow(table, name, py)?; Ok(df) } /// Construct datafusion dataframe from polars #[pyo3(signature = (data, name=None))] pub fn from_polars(&self, data: Bound<'_, PyAny>, name: Option<&str>) -> PyResult { // Convert Polars dataframe to Arrow Table let table = data.call_method0("to_arrow")?; // Convert Arrow Table to datafusion DataFrame let df = self.from_arrow(table, name, data.py())?; Ok(df) } pub fn register_table(&self, name: &str, table: Bound<'_, PyAny>) -> PyDataFusionResult<()> { let session = self.clone().into_bound_py_any(table.py())?; let table = PyTable::new(table, Some(session))?; self.ctx.register_table(name, table.table)?; Ok(()) } pub fn deregister_table(&self, name: &str) -> PyDataFusionResult<()> { self.ctx.deregister_table(name)?; Ok(()) } pub fn register_table_factory( &self, format: &str, mut factory: Bound<'_, PyAny>, ) -> PyDataFusionResult<()> { if factory.hasattr("__datafusion_table_provider_factory__")? { let py = factory.py(); let ffi = self.ffi_logical_codec(); let codec_capsule = create_logical_extension_capsule(py, ffi.as_ref())?; factory = factory .getattr("__datafusion_table_provider_factory__")? .call1((codec_capsule,))?; } let factory: Arc = if let Ok(capsule) = factory.cast::().map_err(py_datafusion_err) { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_table_provider_factory"))? .cast(); let factory = unsafe { data.as_ref() }; factory.into() } else { Arc::new(RustWrappedPyTableProviderFactory::new( factory.into(), self.ffi_logical_codec(), )) }; let st = self.ctx.state_ref(); let mut lock = st.write(); lock.table_factories_mut() .insert(format.to_owned(), factory); Ok(()) } pub fn register_catalog_provider_list( &self, mut provider: Bound, ) -> PyDataFusionResult<()> { if provider.hasattr("__datafusion_catalog_provider_list__")? { let py = provider.py(); let ffi = self.ffi_logical_codec(); let codec_capsule = create_logical_extension_capsule(py, ffi.as_ref())?; provider = provider .getattr("__datafusion_catalog_provider_list__")? .call1((codec_capsule,))?; } let provider = if let Ok(capsule) = provider.cast::() { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_catalog_provider_list"))? .cast(); let provider = unsafe { data.as_ref() }; let provider: Arc = provider.into(); provider } else { match provider.extract::() { Ok(py_catalog_list) => py_catalog_list.catalog_list, Err(_) => Arc::new(RustWrappedPyCatalogProviderList::new( provider.into(), self.ffi_logical_codec(), )) as Arc, } }; self.ctx.register_catalog_list(provider); Ok(()) } pub fn register_catalog_provider( &self, name: &str, mut provider: Bound<'_, PyAny>, ) -> PyDataFusionResult<()> { if provider.hasattr("__datafusion_catalog_provider__")? { let py = provider.py(); let ffi = self.ffi_logical_codec(); let codec_capsule = create_logical_extension_capsule(py, ffi.as_ref())?; provider = provider .getattr("__datafusion_catalog_provider__")? .call1((codec_capsule,))?; } let provider = if let Ok(capsule) = provider.cast::() { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_catalog_provider"))? .cast(); let provider = unsafe { data.as_ref() }; let provider: Arc = provider.into(); provider } else { match provider.extract::() { Ok(py_catalog) => py_catalog.catalog, Err(_) => Arc::new(RustWrappedPyCatalogProvider::new( provider.into(), self.ffi_logical_codec(), )) as Arc, } }; let _ = self.ctx.register_catalog(name, provider); Ok(()) } /// Construct datafusion dataframe from Arrow Table pub fn register_table_provider( &self, name: &str, provider: Bound<'_, PyAny>, ) -> PyDataFusionResult<()> { // Deprecated: use `register_table` instead self.register_table(name, provider) } pub fn register_record_batches( &self, name: &str, partitions: PyArrowType>>, ) -> PyDataFusionResult<()> { let schema = partitions.0[0][0].schema(); let table = MemTable::try_new(schema, partitions.0)?; self.ctx.register_table(name, Arc::new(table))?; Ok(()) } pub fn read_batches( &self, batches: PyArrowType>, ) -> PyDataFusionResult { Ok(PyDataFrame::new(self.ctx.read_batches(batches.0)?)) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = (name, path, table_partition_cols=vec![], parquet_pruning=true, file_extension=".parquet", skip_metadata=true, schema=None, file_sort_order=None))] pub fn register_parquet( &self, name: &str, path: PathBuf, table_partition_cols: Vec<(String, PyArrowType)>, parquet_pruning: bool, file_extension: &str, skip_metadata: bool, schema: Option>, file_sort_order: Option>>, py: Python, ) -> PyDataFusionResult<()> { let options = build_parquet_options( table_partition_cols, parquet_pruning, file_extension, skip_metadata, &schema, file_sort_order, ); wait_for_future( py, self.ctx .register_parquet(name, path_to_str(&path)?, options), )??; Ok(()) } #[pyo3(signature = (name, path, options=None))] pub fn register_csv( &self, name: &str, path: &Bound<'_, PyAny>, options: Option<&PyCsvReadOptions>, py: Python, ) -> PyDataFusionResult<()> { let options = convert_csv_options(options)?; if path.is_instance_of::() { let paths = path .extract::>()? .iter() .map(|p| path_to_str(p).map(str::to_owned)) .collect::>>()?; wait_for_future( py, self.register_csv_from_multiple_paths(name, paths, options), )??; } else { let path = path.extract::()?; wait_for_future( py, self.ctx.register_csv(name, path_to_str(&path)?, options), )??; } Ok(()) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = (name, path, schema=None, schema_infer_max_records=1000, file_extension=".json", table_partition_cols=vec![], file_compression_type=None))] pub fn register_json( &self, name: &str, path: PathBuf, schema: Option>, schema_infer_max_records: usize, file_extension: &str, table_partition_cols: Vec<(String, PyArrowType)>, file_compression_type: Option, py: Python, ) -> PyDataFusionResult<()> { let options = build_json_options( table_partition_cols, file_compression_type, schema_infer_max_records, file_extension, &schema, )?; wait_for_future( py, self.ctx.register_json(name, path_to_str(&path)?, options), )??; Ok(()) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = (name, path, schema=None, file_extension=".avro", table_partition_cols=vec![]))] pub fn register_avro( &self, name: &str, path: PathBuf, schema: Option>, file_extension: &str, table_partition_cols: Vec<(String, PyArrowType)>, py: Python, ) -> PyDataFusionResult<()> { let options = build_avro_options(table_partition_cols, file_extension, &schema); wait_for_future( py, self.ctx.register_avro(name, path_to_str(&path)?, options), )??; Ok(()) } #[pyo3(signature = (name, path, schema=None, file_extension=".arrow", table_partition_cols=vec![]))] pub fn register_arrow( &self, name: &str, path: PathBuf, schema: Option>, file_extension: &str, table_partition_cols: Vec<(String, PyArrowType)>, py: Python, ) -> PyDataFusionResult<()> { let options = build_arrow_options(table_partition_cols, file_extension, &schema); wait_for_future( py, self.ctx.register_arrow(name, path_to_str(&path)?, options), )??; Ok(()) } pub fn register_batch( &self, name: &str, batch: PyArrowType, ) -> PyDataFusionResult<()> { self.ctx.register_batch(name, batch.0)?; Ok(()) } // Registers a PyArrow.Dataset pub fn register_dataset( &self, name: &str, dataset: &Bound<'_, PyAny>, py: Python, ) -> PyDataFusionResult<()> { let table: Arc = Arc::new(Dataset::new(dataset, py)?); self.ctx.register_table(name, table)?; Ok(()) } pub fn register_udf(&self, udf: PyScalarUDF) -> PyResult<()> { self.ctx.register_udf(udf.function); Ok(()) } pub fn deregister_udf(&self, name: &str) { self.ctx.deregister_udf(name); } pub fn register_udaf(&self, udaf: PyAggregateUDF) -> PyResult<()> { self.ctx.register_udaf(udaf.function); Ok(()) } /// Register all `datafusion-spark` UDFs/UDAFs/UDWFs, overriding any built-in /// DataFusion functions of the same name with their Spark-semantics version. pub fn enable_spark_functions(&self) -> PyResult<()> { for udf in datafusion_spark::all_default_scalar_functions() { self.ctx.register_udf((*udf).clone()); } for udaf in datafusion_spark::all_default_aggregate_functions() { self.ctx.register_udaf((*udaf).clone()); } for udwf in datafusion_spark::all_default_window_functions() { self.ctx.register_udwf((*udwf).clone()); } Ok(()) } pub fn deregister_udaf(&self, name: &str) { self.ctx.deregister_udaf(name); } pub fn register_udwf(&self, udwf: PyWindowUDF) -> PyResult<()> { self.ctx.register_udwf(udwf.function); Ok(()) } pub fn deregister_udwf(&self, name: &str) { self.ctx.deregister_udwf(name); } pub fn udf(&self, name: &str) -> PyResult { if !self.ctx.udfs().contains(name) { return Err(PyKeyError::new_err(format!("no UDF named '{name}'"))); } let function = (*self.ctx.udf(name).map_err(py_datafusion_err)?).clone(); Ok(PyScalarUDF { function }) } pub fn udaf(&self, name: &str) -> PyResult { if !self.ctx.udafs().contains(name) { return Err(PyKeyError::new_err(format!("no UDAF named '{name}'"))); } let function = (*self.ctx.udaf(name).map_err(py_datafusion_err)?).clone(); Ok(PyAggregateUDF { function }) } pub fn udwf(&self, name: &str) -> PyResult { if !self.ctx.udwfs().contains(name) { return Err(PyKeyError::new_err(format!("no UDWF named '{name}'"))); } let function = (*self.ctx.udwf(name).map_err(py_datafusion_err)?).clone(); Ok(PyWindowUDF { function }) } pub fn udfs(&self) -> Vec { let mut names: Vec = self.ctx.udfs().into_iter().collect(); names.sort(); names } pub fn udafs(&self) -> Vec { let mut names: Vec = self.ctx.udafs().into_iter().collect(); names.sort(); names } pub fn udwfs(&self) -> Vec { let mut names: Vec = self.ctx.udwfs().into_iter().collect(); names.sort(); names } #[pyo3(signature = (name="datafusion"))] pub fn catalog(&self, py: Python, name: &str) -> PyResult> { let catalog = self.ctx.catalog(name).ok_or(PyKeyError::new_err(format!( "Catalog with name {name} doesn't exist." )))?; match catalog.downcast_ref::() { Some(wrapped_schema) => Ok(wrapped_schema.catalog_provider.clone_ref(py)), None => { Ok(PyCatalog::new_from_parts(catalog, self.ffi_logical_codec()).into_py_any(py)?) } } } pub fn catalog_names(&self) -> HashSet { self.ctx.catalog_names().into_iter().collect() } pub fn table(&self, name: &str, py: Python) -> PyResult { let res = wait_for_future(py, self.ctx.table(name)) .map_err(|e| PyKeyError::new_err(e.to_string()))?; match res { Ok(df) => Ok(PyDataFrame::new(df)), Err(e) => { if let datafusion::error::DataFusionError::Plan(msg) = &e && msg.contains("No table named") { return Err(PyKeyError::new_err(msg.to_string())); } Err(py_datafusion_err(e)) } } } pub fn table_exist(&self, name: &str) -> PyDataFusionResult { Ok(self.ctx.table_exist(name)?) } pub fn empty_table(&self) -> PyDataFusionResult { Ok(PyDataFrame::new(self.ctx.read_empty()?)) } pub fn session_id(&self) -> String { self.ctx.session_id() } /// Return a copy of the active `SessionConfig`. Mutating the returned /// config does not affect this context. pub fn copied_config(&self) -> PySessionConfig { self.ctx.copied_config().into() } /// Parse a string like `"100M"`, `"1.5G"`, or `"512K"` into a byte count. /// `"0"` is accepted and returns 0. Use this when constructing a /// `RuntimeEnvBuilder` from a human-friendly size string. #[staticmethod] pub fn parse_capacity_limit(config_name: &str, limit: &str) -> PyDataFusionResult { Ok(SessionContext::parse_capacity_limit(config_name, limit)?) } pub fn session_start_time(&self) -> String { self.ctx.session_start_time().to_rfc3339() } pub fn enable_ident_normalization(&self) -> bool { self.ctx.enable_ident_normalization() } pub fn parse_sql_expr(&self, sql: &str, schema: PyDFSchema) -> PyDataFusionResult { let df_schema: DFSchema = schema.into(); Ok(self.ctx.parse_sql_expr(sql, &df_schema)?.into()) } pub fn execute_logical_plan( &self, plan: PyLogicalPlan, py: Python, ) -> PyDataFusionResult { let df = wait_for_future( py, self.ctx.execute_logical_plan(plan.plan.as_ref().clone()), )??; Ok(PyDataFrame::new(df)) } pub fn refresh_catalogs(&self, py: Python) -> PyDataFusionResult<()> { wait_for_future(py, self.ctx.refresh_catalogs())??; Ok(()) } pub fn remove_optimizer_rule(&self, name: &str) -> bool { self.ctx.remove_optimizer_rule(name) } pub fn add_physical_optimizer_rule(&self, rule: Bound<'_, PyAny>) -> PyDataFusionResult<()> { let rule = physical_optimizer_rule_from_pycapsule(&rule)?; let state_ref = self.ctx.state_ref(); let mut guard = state_ref.write(); let new_state = SessionStateBuilder::new_from_existing(guard.clone()) .with_physical_optimizer_rule(rule) .build(); *guard = new_state; Ok(()) } pub fn table_provider(&self, name: &str, py: Python) -> PyResult { let provider = wait_for_future(py, self.ctx.table_provider(name)) // Outer error: runtime/async failure .map_err(|e| PyRuntimeError::new_err(e.to_string()))? // Inner error: table not found .map_err(|e| PyKeyError::new_err(e.to_string()))?; Ok(PyTable { table: provider }) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = (path, schema=None, schema_infer_max_records=1000, file_extension=".json", table_partition_cols=vec![], file_compression_type=None))] pub fn read_json( &self, path: PathBuf, schema: Option>, schema_infer_max_records: usize, file_extension: &str, table_partition_cols: Vec<(String, PyArrowType)>, file_compression_type: Option, py: Python, ) -> PyDataFusionResult { let options = build_json_options( table_partition_cols, file_compression_type, schema_infer_max_records, file_extension, &schema, )?; let df = wait_for_future(py, self.ctx.read_json(path_to_str(&path)?, options))??; Ok(PyDataFrame::new(df)) } #[pyo3(signature = ( path, options=None))] pub fn read_csv( &self, path: &Bound<'_, PyAny>, options: Option<&PyCsvReadOptions>, py: Python, ) -> PyDataFusionResult { let options = convert_csv_options(options)?; let paths: Vec = if path.is_instance_of::() { path.extract::>()? .iter() .map(|p| path_to_str(p).map(str::to_owned)) .collect::>()? } else { vec![path_to_str(&path.extract::()?)?.to_owned()] }; let df = wait_for_future(py, self.ctx.read_csv(paths, options))??; Ok(PyDataFrame::new(df)) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = ( path, table_partition_cols=vec![], parquet_pruning=true, file_extension=".parquet", skip_metadata=true, schema=None, file_sort_order=None))] pub fn read_parquet( &self, path: PathBuf, table_partition_cols: Vec<(String, PyArrowType)>, parquet_pruning: bool, file_extension: &str, skip_metadata: bool, schema: Option>, file_sort_order: Option>>, py: Python, ) -> PyDataFusionResult { let options = build_parquet_options( table_partition_cols, parquet_pruning, file_extension, skip_metadata, &schema, file_sort_order, ); let df = PyDataFrame::new(wait_for_future( py, self.ctx.read_parquet(path_to_str(&path)?, options), )??); Ok(df) } #[allow(clippy::too_many_arguments)] #[pyo3(signature = (path, schema=None, table_partition_cols=vec![], file_extension=".avro"))] pub fn read_avro( &self, path: PathBuf, schema: Option>, table_partition_cols: Vec<(String, PyArrowType)>, file_extension: &str, py: Python, ) -> PyDataFusionResult { let options = build_avro_options(table_partition_cols, file_extension, &schema); let df = wait_for_future(py, self.ctx.read_avro(path_to_str(&path)?, options))??; Ok(PyDataFrame::new(df)) } #[pyo3(signature = (path, schema=None, file_extension=".arrow", table_partition_cols=vec![]))] pub fn read_arrow( &self, path: PathBuf, schema: Option>, file_extension: &str, table_partition_cols: Vec<(String, PyArrowType)>, py: Python, ) -> PyDataFusionResult { let options = build_arrow_options(table_partition_cols, file_extension, &schema); let df = wait_for_future(py, self.ctx.read_arrow(path_to_str(&path)?, options))??; Ok(PyDataFrame::new(df)) } pub fn read_table(&self, table: Bound<'_, PyAny>) -> PyDataFusionResult { let session = self.clone().into_bound_py_any(table.py())?; let table = PyTable::new(table, Some(session))?; let df = self.ctx.read_table(table.table())?; Ok(PyDataFrame::new(df)) } fn __repr__(&self) -> PyResult { let config = self.ctx.copied_config(); let mut config_entries = config .options() .entries() .iter() .filter(|e| e.value.is_some()) .map(|e| format!("{} = {}", e.key, e.value.as_ref().unwrap())) .collect::>(); config_entries.sort(); Ok(format!( "SessionContext: id={}; configs=[\n\t{}]", self.session_id(), config_entries.join("\n\t") )) } /// Execute a partition of an execution plan and return a stream of record batches pub fn execute( &self, plan: PyExecutionPlan, part: usize, py: Python, ) -> PyDataFusionResult { let ctx: TaskContext = TaskContext::from(&self.ctx.state()); let plan = plan.plan.clone(); let stream = spawn_future(py, async move { plan.execute(part, Arc::new(ctx)) })?; Ok(PyRecordBatchStream::new(stream)) } pub fn __datafusion_task_context_provider__<'py>( &self, py: Python<'py>, ) -> PyResult> { let name = cr"datafusion_task_context_provider".into(); let ctx_provider = Arc::clone(&self.ctx) as Arc; let ffi_ctx_provider = FFI_TaskContextProvider::from(&ctx_provider); PyCapsule::new(py, ffi_ctx_provider, Some(name)) } pub fn __datafusion_logical_extension_codec__<'py>( &self, py: Python<'py>, ) -> PyResult> { let ffi = self.ffi_logical_codec(); create_logical_extension_capsule(py, ffi.as_ref()) } pub fn with_logical_extension_codec<'py>( &self, codec: Bound<'py, PyAny>, ) -> PyDataFusionResult { let inner_ffi = ffi_logical_codec_from_pycapsule(codec)?; let inner: Arc = (&inner_ffi).into(); let logical_codec = Arc::new(PythonLogicalCodec::new(inner)); Ok(Self { ctx: Arc::clone(&self.ctx), logical_codec, physical_codec: Arc::clone(&self.physical_codec), }) } pub fn __datafusion_physical_extension_codec__<'py>( &self, py: Python<'py>, ) -> PyResult> { let ffi = self.ffi_physical_codec(); create_physical_extension_capsule(py, ffi.as_ref()) } pub fn with_physical_extension_codec<'py>( &self, codec: Bound<'py, PyAny>, ) -> PyDataFusionResult { let inner = physical_codec_from_pycapsule(&codec)?; let physical_codec = Arc::new(PythonPhysicalCodec::new(inner)); Ok(Self { ctx: Arc::clone(&self.ctx), logical_codec: Arc::clone(&self.logical_codec), physical_codec, }) } pub fn with_python_udf_inlining(&self, enabled: bool) -> Self { let logical_codec = Arc::new( PythonLogicalCodec::new(Arc::clone(self.logical_codec.inner())) .with_python_udf_inlining(enabled), ); let physical_codec = Arc::new( PythonPhysicalCodec::new(Arc::clone(self.physical_codec.inner())) .with_python_udf_inlining(enabled), ); Self { ctx: Arc::clone(&self.ctx), logical_codec, physical_codec, } } } impl PySessionContext { async fn _table(&self, name: &str) -> datafusion::common::Result { self.ctx.table(name).await } async fn register_csv_from_multiple_paths( &self, name: &str, table_paths: Vec, options: CsvReadOptions<'_>, ) -> datafusion::common::Result<()> { let table_paths = table_paths.to_urls()?; let session_config = self.ctx.copied_config(); let listing_options = options.to_listing_options(&session_config, self.ctx.copied_table_options()); let option_extension = listing_options.file_extension.clone(); if table_paths.is_empty() { return exec_err!("No table paths were provided"); } // check if the file extension matches the expected extension for path in &table_paths { let file_path = path.as_str(); if !file_path.ends_with(option_extension.as_str()) && !path.is_collection() { return exec_err!( "File path '{file_path}' does not match the expected extension '{option_extension}'" ); } } let resolved_schema = options .get_resolved_schema(&session_config, self.ctx.state(), table_paths[0].clone()) .await?; let config = ListingTableConfig::new_with_multi_paths(table_paths) .with_listing_options(listing_options) .with_schema(resolved_schema); let table = ListingTable::try_new(config)?; self.ctx .register_table(TableReference::Bare { table: name.into() }, Arc::new(table))?; Ok(()) } /// Session-scoped logical codec. Sibling modules read this when they /// need to serialize/deserialize logical-layer types (LogicalPlan, /// Expr) against the user-installed (or default) codec stack. pub(crate) fn logical_codec(&self) -> &Arc { &self.logical_codec } /// Session-scoped physical codec. Sibling modules read this for /// ExecutionPlan / PhysicalExpr serialization. pub(crate) fn physical_codec(&self) -> &Arc { &self.physical_codec } /// Build an FFI-wrapped clone of the session's logical codec on demand. /// Used at every site that exports the codec across an FFI boundary /// (capsule getters, Rust wrappers for Python-defined providers, etc.). pub(crate) fn ffi_logical_codec(&self) -> Arc { let inner: Arc = Arc::clone(&self.logical_codec) as Arc; let runtime = get_tokio_runtime().handle().clone(); let ctx_provider = Arc::clone(&self.ctx) as Arc; Arc::new(FFI_LogicalExtensionCodec::new( inner, Some(runtime), &ctx_provider, )) } /// Build an FFI-wrapped clone of the session's physical codec on demand. pub(crate) fn ffi_physical_codec(&self) -> Arc { let inner: Arc = Arc::clone(&self.physical_codec) as Arc; let runtime = get_tokio_runtime().handle().clone(); let ctx_provider = Arc::clone(&self.ctx) as Arc; Arc::new(FFI_PhysicalExtensionCodec::new( inner, Some(runtime), &ctx_provider, )) } } pub fn parse_file_compression_type( file_compression_type: Option, ) -> Result { FileCompressionType::from_str(&file_compression_type.unwrap_or_default()).map_err(|_| { PyValueError::new_err("file_compression_type must be one of: gzip, bz2, xz, zstd") }) } fn path_to_str(path: &Path) -> PyDataFusionResult<&str> { path.to_str() .ok_or_else(|| PyValueError::new_err("Unable to convert path to a string").into()) } fn convert_csv_options( options: Option<&PyCsvReadOptions>, ) -> PyDataFusionResult> { Ok(options .map(|opts| opts.try_into()) .transpose()? .unwrap_or_default()) } fn convert_partition_cols( table_partition_cols: Vec<(String, PyArrowType)>, ) -> Vec<(String, DataType)> { table_partition_cols .into_iter() .map(|(name, ty)| (name, ty.0)) .collect() } fn convert_file_sort_order( file_sort_order: Option>>, ) -> Vec> { file_sort_order .unwrap_or_default() .into_iter() .map(|e| e.into_iter().map(|f| f.into()).collect()) .collect() } fn build_parquet_options<'a>( table_partition_cols: Vec<(String, PyArrowType)>, parquet_pruning: bool, file_extension: &'a str, skip_metadata: bool, schema: &'a Option>, file_sort_order: Option>>, ) -> ParquetReadOptions<'a> { let mut options = ParquetReadOptions::default() .table_partition_cols(convert_partition_cols(table_partition_cols)) .parquet_pruning(parquet_pruning) .skip_metadata(skip_metadata); options.file_extension = file_extension; options.schema = schema.as_ref().map(|x| &x.0); options.file_sort_order = convert_file_sort_order(file_sort_order); options } fn build_json_options<'a>( table_partition_cols: Vec<(String, PyArrowType)>, file_compression_type: Option, schema_infer_max_records: usize, file_extension: &'a str, schema: &'a Option>, ) -> Result, PyErr> { let mut options = JsonReadOptions::default() .table_partition_cols(convert_partition_cols(table_partition_cols)) .file_compression_type(parse_file_compression_type(file_compression_type)?); options.schema_infer_max_records = schema_infer_max_records; options.file_extension = file_extension; options.schema = schema.as_ref().map(|x| &x.0); Ok(options) } fn build_arrow_options<'a>( table_partition_cols: Vec<(String, PyArrowType)>, file_extension: &'a str, schema: &'a Option>, ) -> ArrowReadOptions<'a> { let mut options = ArrowReadOptions::default() .table_partition_cols(convert_partition_cols(table_partition_cols)); options.file_extension = file_extension; options.schema = schema.as_ref().map(|x| &x.0); options } fn build_avro_options<'a>( table_partition_cols: Vec<(String, PyArrowType)>, file_extension: &'a str, schema: &'a Option>, ) -> AvroReadOptions<'a> { let mut options = AvroReadOptions::default() .table_partition_cols(convert_partition_cols(table_partition_cols)); options.file_extension = file_extension; options.schema = schema.as_ref().map(|x| &x.0); options } impl From for SessionContext { fn from(ctx: PySessionContext) -> SessionContext { ctx.ctx.as_ref().clone() } } impl From for PySessionContext { fn from(ctx: SessionContext) -> PySessionContext { PySessionContext { ctx: Arc::new(ctx), logical_codec: Arc::new(PythonLogicalCodec::default()), physical_codec: Arc::new(PythonPhysicalCodec::default()), } } } apache-datafusion-python-54.0.0/crates/core/src/dataframe.rs000066400000000000000000001600431521727353300240010ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::ffi::{CStr, CString}; use std::ptr::NonNull; use std::str::FromStr; use std::sync::Arc; use arrow::array::{Array, ArrayRef, RecordBatch, RecordBatchReader, new_null_array}; use arrow::compute::can_cast_types; use arrow::error::ArrowError; use arrow::ffi::FFI_ArrowSchema; use arrow::ffi_stream::FFI_ArrowArrayStream; use arrow::pyarrow::FromPyArrow; use cstr::cstr; use datafusion::arrow::datatypes::{Schema, SchemaRef}; use datafusion::arrow::pyarrow::{PyArrowType, ToPyArrow}; use datafusion::arrow::util::pretty; use datafusion::catalog::TableProvider; use datafusion::common::UnnestOptions; use datafusion::config::{CsvOptions, ParquetColumnOptions, ParquetOptions, TableParquetOptions}; use datafusion::dataframe::{DataFrame, DataFrameWriteOptions}; use datafusion::error::DataFusionError; use datafusion::execution::SendableRecordBatchStream; use datafusion::execution::context::TaskContext; use datafusion::logical_expr::dml::InsertOp; use datafusion::logical_expr::{LogicalPlan, SortExpr}; use datafusion::parquet::basic::{BrotliLevel, Compression, GzipLevel, ZstdLevel}; use datafusion::physical_plan::{ ExecutionPlan as DFExecutionPlan, collect as df_collect, collect_partitioned as df_collect_partitioned, execute_stream as df_execute_stream, execute_stream_partitioned as df_execute_stream_partitioned, }; use datafusion::prelude::*; use datafusion_python_util::{is_ipython_env, spawn_future, wait_for_future}; use futures::{StreamExt, TryStreamExt}; use parking_lot::Mutex; use pyo3::PyErr; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; use pyo3::pybacked::PyBackedStr; use pyo3::types::{PyCapsule, PyList, PyTuple, PyTupleMethods}; use crate::common::data_type::PyScalarValue; use crate::errors::{PyDataFusionError, PyDataFusionResult, py_datafusion_err}; use crate::expr::PyExpr; use crate::expr::sort_expr::{PySortExpr, to_sort_expressions}; use crate::physical_plan::PyExecutionPlan; use crate::record_batch::{PyRecordBatchStream, poll_next_batch}; use crate::sql::logical::PyLogicalPlan; use crate::table::{PyTable, TempViewTable}; /// File-level static CStr for the Arrow array stream capsule name. static ARROW_ARRAY_STREAM_NAME: &CStr = cstr!("arrow_array_stream"); // Type aliases to simplify very complex types used in this file and // avoid compiler complaints about deeply nested types in struct fields. type CachedBatches = Option<(Vec, bool)>; type SharedCachedBatches = Arc>; /// Configuration for DataFrame display formatting #[derive(Debug, Clone)] pub struct FormatterConfig { /// Maximum memory in bytes to use for display (default: 2MB) pub max_bytes: usize, /// Minimum number of rows to display (default: 10) pub min_rows: usize, /// Maximum number of rows to include in __repr__ output (default: 10) pub max_rows: usize, } impl Default for FormatterConfig { fn default() -> Self { Self { max_bytes: 2 * 1024 * 1024, // 2MB min_rows: 10, max_rows: 10, } } } impl FormatterConfig { /// Validates that all configuration values are positive integers. /// /// # Returns /// /// `Ok(())` if all values are valid, or an `Err` with a descriptive error message. pub fn validate(&self) -> Result<(), String> { if self.max_bytes == 0 { return Err("max_bytes must be a positive integer".to_string()); } if self.min_rows == 0 { return Err("min_rows must be a positive integer".to_string()); } if self.max_rows == 0 { return Err("max_rows must be a positive integer".to_string()); } if self.min_rows > self.max_rows { return Err("min_rows must be less than or equal to max_rows".to_string()); } Ok(()) } } /// Holds the Python formatter and its configuration struct PythonFormatter<'py> { /// The Python formatter object formatter: Bound<'py, PyAny>, /// The formatter configuration config: FormatterConfig, } /// Get the Python formatter and its configuration fn get_python_formatter_with_config(py: Python) -> PyResult { let formatter = import_python_formatter(py)?; let config = build_formatter_config_from_python(&formatter)?; Ok(PythonFormatter { formatter, config }) } /// Get the Python formatter from the datafusion.dataframe_formatter module fn import_python_formatter(py: Python<'_>) -> PyResult> { let formatter_module = py.import("datafusion.dataframe_formatter")?; let get_formatter = formatter_module.getattr("get_formatter")?; get_formatter.call0() } // Helper function to extract attributes with fallback to default fn get_attr<'a, T>(py_object: &'a Bound<'a, PyAny>, attr_name: &str, default_value: T) -> T where T: for<'py> pyo3::FromPyObject<'py, 'py> + Clone, { py_object .getattr(attr_name) .and_then(|v| v.extract::().map_err(Into::::into)) .unwrap_or_else(|_| default_value.clone()) } /// Helper function to create a FormatterConfig from a Python formatter object fn build_formatter_config_from_python(formatter: &Bound<'_, PyAny>) -> PyResult { let default_config = FormatterConfig::default(); let max_bytes = get_attr(formatter, "max_memory_bytes", default_config.max_bytes); let min_rows = get_attr(formatter, "min_rows", default_config.min_rows); // Backward compatibility: Try max_rows first (new name), fall back to repr_rows (deprecated), // then use default. This ensures backward compatibility with custom formatter implementations // during the deprecation period. let max_rows = get_attr(formatter, "max_rows", 0usize); let max_rows = if max_rows > 0 { // max_rows attribute exists and has a value max_rows } else { // Try the deprecated repr_rows attribute let repr_rows = get_attr(formatter, "repr_rows", 0usize); if repr_rows > 0 { repr_rows } else { // Use default default_config.max_rows } }; let config = FormatterConfig { max_bytes, min_rows, max_rows, }; // Return the validated config, converting String error to PyErr config.validate().map_err(PyValueError::new_err)?; Ok(config) } /// Python mapping of `ParquetOptions` (includes just the writer-related options). #[pyclass( from_py_object, frozen, name = "ParquetWriterOptions", module = "datafusion", subclass )] #[derive(Clone, Default)] pub struct PyParquetWriterOptions { options: ParquetOptions, } #[pymethods] impl PyParquetWriterOptions { #[new] #[allow(clippy::too_many_arguments)] pub fn new( data_pagesize_limit: usize, write_batch_size: usize, writer_version: &str, skip_arrow_metadata: bool, compression: Option, dictionary_enabled: Option, dictionary_page_size_limit: usize, statistics_enabled: Option, max_row_group_size: usize, created_by: String, column_index_truncate_length: Option, statistics_truncate_length: Option, data_page_row_count_limit: usize, encoding: Option, bloom_filter_on_write: bool, bloom_filter_fpp: Option, bloom_filter_ndv: Option, allow_single_file_parallelism: bool, maximum_parallel_row_group_writers: usize, maximum_buffered_record_batches_per_stream: usize, ) -> PyResult { let writer_version = datafusion::common::parquet_config::DFParquetWriterVersion::from_str(writer_version) .map_err(py_datafusion_err)?; Ok(Self { options: ParquetOptions { data_pagesize_limit, write_batch_size, writer_version, skip_arrow_metadata, compression, dictionary_enabled, dictionary_page_size_limit, statistics_enabled, max_row_group_size, created_by, column_index_truncate_length, statistics_truncate_length, data_page_row_count_limit, encoding, bloom_filter_on_write, bloom_filter_fpp, bloom_filter_ndv, allow_single_file_parallelism, maximum_parallel_row_group_writers, maximum_buffered_record_batches_per_stream, ..Default::default() }, }) } } /// Python mapping of `ParquetColumnOptions`. #[pyclass( from_py_object, frozen, name = "ParquetColumnOptions", module = "datafusion", subclass )] #[derive(Clone, Default)] pub struct PyParquetColumnOptions { options: ParquetColumnOptions, } #[pymethods] impl PyParquetColumnOptions { #[new] pub fn new( bloom_filter_enabled: Option, encoding: Option, dictionary_enabled: Option, compression: Option, statistics_enabled: Option, bloom_filter_fpp: Option, bloom_filter_ndv: Option, ) -> Self { Self { options: ParquetColumnOptions { bloom_filter_enabled, encoding, dictionary_enabled, compression, statistics_enabled, bloom_filter_fpp, bloom_filter_ndv, }, } } } /// A PyDataFrame is a representation of a logical plan and an API to compose statements. /// Use it to build a plan and `.collect()` to execute the plan and collect the result. /// The actual execution of a plan runs natively on Rust and Arrow on a multi-threaded environment. #[pyclass( from_py_object, name = "DataFrame", module = "datafusion", subclass, frozen )] #[derive(Clone)] pub struct PyDataFrame { df: Arc, // In IPython environment cache batches between __repr__ and _repr_html_ calls. batches: SharedCachedBatches, // Cache the last physical plan so that metrics are available after execution. last_plan: Arc>>>, } impl PyDataFrame { /// creates a new PyDataFrame pub fn new(df: DataFrame) -> Self { Self { df: Arc::new(df), batches: Arc::new(Mutex::new(None)), last_plan: Arc::new(Mutex::new(None)), } } /// Return a clone of the inner Arc for crate-local callers. pub(crate) fn inner_df(&self) -> Arc { Arc::clone(&self.df) } fn prepare_repr_string<'py>( &self, py: Python<'py>, as_html: bool, ) -> PyDataFusionResult { // Get the Python formatter and config let PythonFormatter { formatter, config } = get_python_formatter_with_config(py)?; let is_ipython = *is_ipython_env(py); let (cached_batches, should_cache) = { let mut cache = self.batches.lock(); let should_cache = is_ipython && cache.is_none(); let batches = cache.take(); (batches, should_cache) }; let (batches, has_more) = match cached_batches { Some(b) => b, None => wait_for_future( py, collect_record_batches_to_display(self.df.as_ref().clone(), config), )??, }; if batches.is_empty() { // This should not be reached, but do it for safety since we index into the vector below return Ok("No data to display".to_string()); } let table_uuid = uuid::Uuid::new_v4().to_string(); // Convert record batches to Py list let py_batches = batches .iter() .map(|rb| rb.to_pyarrow(py)) .collect::>>>()?; let py_schema = self.schema().into_pyobject(py)?; let kwargs = pyo3::types::PyDict::new(py); let py_batches_list = PyList::new(py, py_batches.as_slice())?; kwargs.set_item("batches", py_batches_list)?; kwargs.set_item("schema", py_schema)?; kwargs.set_item("has_more", has_more)?; kwargs.set_item("table_uuid", table_uuid)?; let method_name = match as_html { true => "format_html", false => "format_str", }; let html_result = formatter.call_method(method_name, (), Some(&kwargs))?; let html_str: String = html_result.extract()?; if should_cache { let mut cache = self.batches.lock(); *cache = Some((batches.clone(), has_more)); } Ok(html_str) } /// Create the physical plan, cache it in `last_plan`, and return the plan together /// with a task context. Centralises the repeated three-line pattern that appears in /// `collect`, `collect_partitioned`, `execute_stream`, and `execute_stream_partitioned`. fn create_and_cache_plan( &self, py: Python, ) -> PyDataFusionResult<(Arc, Arc)> { let df = self.df.as_ref().clone(); let new_plan = wait_for_future(py, df.create_physical_plan())??; *self.last_plan.lock() = Some(Arc::clone(&new_plan)); let task_ctx = Arc::new(self.df.as_ref().task_ctx()); Ok((new_plan, task_ctx)) } async fn collect_column_inner(&self, column: &str) -> Result { let batches = self .df .as_ref() .clone() .select_columns(&[column])? .collect() .await?; let arrays = batches .iter() .map(|b| b.column(0).as_ref()) .collect::>(); arrow_select::concat::concat(&arrays).map_err(Into::into) } } /// Synchronous wrapper around partitioned [`SendableRecordBatchStream`]s used /// for the `__arrow_c_stream__` implementation. /// /// It drains each partition's stream sequentially, yielding record batches in /// their original partition order. When a `projection` is set, each batch is /// converted via `record_batch_into_schema` to apply schema changes per batch. struct PartitionedDataFrameStreamReader { streams: Vec, schema: SchemaRef, projection: Option, current: usize, } impl Iterator for PartitionedDataFrameStreamReader { type Item = Result; fn next(&mut self) -> Option { while self.current < self.streams.len() { let stream = &mut self.streams[self.current]; let fut = poll_next_batch(stream); let result = Python::attach(|py| wait_for_future(py, fut)); match result { Ok(Ok(Some(batch))) => { let batch = if let Some(ref schema) = self.projection { match record_batch_into_schema(batch, schema.as_ref()) { Ok(b) => b, Err(e) => return Some(Err(e)), } } else { batch }; return Some(Ok(batch)); } Ok(Ok(None)) => { self.current += 1; continue; } Ok(Err(e)) => { return Some(Err(ArrowError::ExternalError(Box::new(e)))); } Err(e) => { return Some(Err(ArrowError::ExternalError(Box::new(e)))); } } } None } } impl RecordBatchReader for PartitionedDataFrameStreamReader { fn schema(&self) -> SchemaRef { self.schema.clone() } } #[pymethods] impl PyDataFrame { /// Enable selection for `df[col]`, `df[col1, col2, col3]`, and `df[[col1, col2, col3]]` fn __getitem__(&self, key: Bound<'_, PyAny>) -> PyDataFusionResult { if let Ok(key) = key.extract::() { // df[col] self.select_exprs(vec![key]) } else if let Ok(tuple) = key.cast::() { // df[col1, col2, col3] let keys = tuple .iter() .map(|item| item.extract::()) .collect::>>()?; self.select_exprs(keys) } else if let Ok(keys) = key.extract::>() { // df[[col1, col2, col3]] self.select_exprs(keys) } else { let message = "DataFrame can only be indexed by string index or indices".to_string(); Err(PyDataFusionError::Common(message)) } } fn __repr__(&self, py: Python) -> PyDataFusionResult { self.prepare_repr_string(py, false) } #[staticmethod] #[expect(unused_variables)] fn default_str_repr<'py>( batches: Vec>, schema: &Bound<'py, PyAny>, has_more: bool, table_uuid: &str, ) -> PyResult { let batches = batches .into_iter() .map(|batch| RecordBatch::from_pyarrow_bound(&batch)) .collect::>>()? .into_iter() .filter(|batch| batch.num_rows() > 0) .collect::>(); if batches.is_empty() { return Ok("No data to display".to_owned()); } let batches_as_displ = pretty::pretty_format_batches(&batches).map_err(py_datafusion_err)?; let additional_str = match has_more { true => "\nData truncated.", false => "", }; Ok(format!("DataFrame()\n{batches_as_displ}{additional_str}")) } fn _repr_html_(&self, py: Python) -> PyDataFusionResult { self.prepare_repr_string(py, true) } /// Calculate summary statistics for a DataFrame fn describe(&self, py: Python) -> PyDataFusionResult { let df = self.df.as_ref().clone(); let stat_df = wait_for_future(py, df.describe())??; Ok(Self::new(stat_df)) } /// Returns the schema from the logical plan fn schema(&self) -> PyArrowType { PyArrowType(self.df.schema().as_arrow().clone()) } /// Convert this DataFrame into a Table Provider that can be used in register_table /// By convention, into_... methods consume self and return the new object. /// Disabling the clippy lint, so we can use &self /// because we're working with Python bindings /// where objects are shared #[allow(clippy::wrong_self_convention)] pub fn into_view(&self, temporary: bool) -> PyDataFusionResult { let table_provider = if temporary { Arc::new(TempViewTable::new(Arc::clone(&self.df))) as Arc } else { // Call the underlying Rust DataFrame::into_view method. // Note that the Rust method consumes self; here we clone the inner Arc // so that we don't invalidate this PyDataFrame. self.df.as_ref().clone().into_view() }; Ok(PyTable::from(table_provider)) } fn alias(&self, alias: &str) -> PyDataFusionResult { let df = self.df.as_ref().clone().alias(alias)?; Ok(Self::new(df)) } #[pyo3(signature = (*args))] fn select_exprs(&self, args: Vec) -> PyDataFusionResult { let args = args.iter().map(|s| s.as_ref()).collect::>(); let df = self.df.as_ref().clone().select_exprs(&args)?; Ok(Self::new(df)) } #[pyo3(signature = (*args))] fn select(&self, args: Vec) -> PyDataFusionResult { let expr: Vec = args.into_iter().map(|e| e.into()).collect(); let df = self.df.as_ref().clone().select(expr)?; Ok(Self::new(df)) } #[pyo3(signature = (*args))] fn drop(&self, args: Vec) -> PyDataFusionResult { let cols = args.iter().map(|s| s.as_ref()).collect::>(); let df = self.df.as_ref().clone().drop_columns(&cols)?; Ok(Self::new(df)) } /// Apply window function expressions to the DataFrame #[pyo3(signature = (*exprs))] fn window(&self, exprs: Vec) -> PyDataFusionResult { let window_exprs = exprs.into_iter().map(|e| e.into()).collect(); let df = self.df.as_ref().clone().window(window_exprs)?; Ok(Self::new(df)) } fn filter(&self, predicate: PyExpr) -> PyDataFusionResult { let df = self.df.as_ref().clone().filter(predicate.into())?; Ok(Self::new(df)) } fn parse_sql_expr(&self, expr: PyBackedStr) -> PyDataFusionResult { self.df .as_ref() .parse_sql_expr(&expr) .map(PyExpr::from) .map_err(PyDataFusionError::from) } fn with_column(&self, name: &str, expr: PyExpr) -> PyDataFusionResult { let df = self.df.as_ref().clone().with_column(name, expr.into())?; Ok(Self::new(df)) } fn with_columns(&self, exprs: Vec) -> PyDataFusionResult { let mut df = self.df.as_ref().clone(); for expr in exprs { let expr: Expr = expr.into(); let name = format!("{}", expr.schema_name()); df = df.with_column(name.as_str(), expr)? } Ok(Self::new(df)) } /// Rename one column by applying a new projection. This is a no-op if the column to be /// renamed does not exist. fn with_column_renamed(&self, old_name: &str, new_name: &str) -> PyDataFusionResult { let df = self .df .as_ref() .clone() .with_column_renamed(old_name, new_name)?; Ok(Self::new(df)) } fn aggregate(&self, group_by: Vec, aggs: Vec) -> PyDataFusionResult { let group_by = group_by.into_iter().map(|e| e.into()).collect(); let aggs = aggs.into_iter().map(|e| e.into()).collect(); let df = self.df.as_ref().clone().aggregate(group_by, aggs)?; Ok(Self::new(df)) } #[pyo3(signature = (*exprs))] fn sort(&self, exprs: Vec) -> PyDataFusionResult { let exprs = to_sort_expressions(exprs); let df = self.df.as_ref().clone().sort(exprs)?; Ok(Self::new(df)) } #[pyo3(signature = (count, offset=0))] fn limit(&self, count: usize, offset: usize) -> PyDataFusionResult { let df = self.df.as_ref().clone().limit(offset, Some(count))?; Ok(Self::new(df)) } /// Executes the plan, returning a list of `RecordBatch`es. /// Unless some order is specified in the plan, there is no /// guarantee of the order of the result. fn collect<'py>(&self, py: Python<'py>) -> PyResult>> { let (plan, task_ctx) = self.create_and_cache_plan(py)?; let batches = wait_for_future(py, df_collect(plan, task_ctx))?.map_err(PyDataFusionError::from)?; // cannot use PyResult> return type due to // https://github.com/PyO3/pyo3/issues/1813 batches.into_iter().map(|rb| rb.to_pyarrow(py)).collect() } /// Cache DataFrame. fn cache(&self, py: Python) -> PyDataFusionResult { let df = wait_for_future(py, self.df.as_ref().clone().cache())??; Ok(Self::new(df)) } /// Executes this DataFrame and collects all results into a vector of vector of RecordBatch /// maintaining the input partitioning. fn collect_partitioned<'py>(&self, py: Python<'py>) -> PyResult>>> { let (plan, task_ctx) = self.create_and_cache_plan(py)?; let batches = wait_for_future(py, df_collect_partitioned(plan, task_ctx))? .map_err(PyDataFusionError::from)?; batches .into_iter() .map(|rbs| rbs.into_iter().map(|rb| rb.to_pyarrow(py)).collect()) .collect() } fn collect_column<'py>(&self, py: Python<'py>, column: &str) -> PyResult> { wait_for_future(py, self.collect_column_inner(column))? .map_err(PyDataFusionError::from)? .to_data() .to_pyarrow(py) } /// Print the result, 20 lines by default #[pyo3(signature = (num=20))] fn show(&self, py: Python, num: usize) -> PyDataFusionResult<()> { let mut df = self.df.as_ref().clone(); df = match self.df.logical_plan() { LogicalPlan::Explain(_) | LogicalPlan::Analyze(_) => { // Explain and Analyzer require they are at the top // of the plan, so do not add a limit. df } _ => df.limit(0, Some(num))?, }; print_dataframe(py, df) } /// Filter out duplicate rows fn distinct(&self) -> PyDataFusionResult { let df = self.df.as_ref().clone().distinct()?; Ok(Self::new(df)) } fn join( &self, right: PyDataFrame, how: &str, left_on: Vec, right_on: Vec, coalesce_keys: bool, ) -> PyDataFusionResult { let join_type = match how { "inner" => JoinType::Inner, "left" => JoinType::Left, "right" => JoinType::Right, "full" => JoinType::Full, "semi" => JoinType::LeftSemi, "anti" => JoinType::LeftAnti, how => { return Err(PyDataFusionError::Common(format!( "The join type {how} does not exist or is not implemented" ))); } }; let left_keys = left_on.iter().map(|s| s.as_ref()).collect::>(); let right_keys = right_on.iter().map(|s| s.as_ref()).collect::>(); let mut df = self.df.as_ref().clone().join( right.df.as_ref().clone(), join_type, &left_keys, &right_keys, None, )?; if coalesce_keys { let mutual_keys = left_keys .iter() .zip(right_keys.iter()) .filter(|(l, r)| l == r) .map(|(key, _)| *key) .collect::>(); let fields_to_coalesce = mutual_keys .iter() .map(|name| { let qualified_fields = df .logical_plan() .schema() .qualified_fields_with_unqualified_name(name); (*name, qualified_fields) }) .filter(|(_, fields)| fields.len() == 2) .collect::>(); let expr: Vec = df .logical_plan() .schema() .fields() .into_iter() .enumerate() .map(|(idx, _)| df.logical_plan().schema().qualified_field(idx)) .filter_map(|(qualifier, field)| { if let Some((key_name, qualified_fields)) = fields_to_coalesce .iter() .find(|(_, qf)| qf.contains(&(qualifier, field))) { // Only add the coalesce expression once (when we encounter the first field) // Skip the second field (it's already included in to coalesce) if (qualifier, field) == qualified_fields[0] { let left_col = Expr::Column(Column::from(qualified_fields[0])); let right_col = Expr::Column(Column::from(qualified_fields[1])); return Some(coalesce(vec![left_col, right_col]).alias(*key_name)); } None } else { Some(Expr::Column(Column::from((qualifier, field)))) } }) .collect(); df = df.select(expr)?; } Ok(Self::new(df)) } fn join_on( &self, right: PyDataFrame, on_exprs: Vec, how: &str, ) -> PyDataFusionResult { let join_type = match how { "inner" => JoinType::Inner, "left" => JoinType::Left, "right" => JoinType::Right, "full" => JoinType::Full, "semi" => JoinType::LeftSemi, "anti" => JoinType::LeftAnti, how => { return Err(PyDataFusionError::Common(format!( "The join type {how} does not exist or is not implemented" ))); } }; let exprs: Vec = on_exprs.into_iter().map(|e| e.into()).collect(); let df = self .df .as_ref() .clone() .join_on(right.df.as_ref().clone(), join_type, exprs)?; Ok(Self::new(df)) } /// Print the query plan #[pyo3(signature = (verbose=false, analyze=false, format=None))] fn explain( &self, py: Python, verbose: bool, analyze: bool, format: Option<&str>, ) -> PyDataFusionResult<()> { let explain_format = match format { Some(f) => f .parse::() .map_err(|e| { PyDataFusionError::Common(format!("Invalid explain format '{}': {}", f, e)) })?, None => datafusion::common::format::ExplainFormat::Indent, }; let opts = datafusion::logical_expr::ExplainOption::default() .with_verbose(verbose) .with_analyze(analyze) .with_format(explain_format); let df = self.df.as_ref().clone().explain_with_options(opts)?; print_dataframe(py, df) } /// Get the logical plan for this `DataFrame` fn logical_plan(&self) -> PyResult { Ok(self.df.as_ref().clone().logical_plan().clone().into()) } /// Get the optimized logical plan for this `DataFrame` fn optimized_logical_plan(&self) -> PyDataFusionResult { Ok(self.df.as_ref().clone().into_optimized_plan()?.into()) } /// Get the execution plan for this `DataFrame` /// /// If the DataFrame has already been executed (e.g. via `collect()`), /// returns the cached plan which includes populated metrics. fn execution_plan(&self, py: Python) -> PyDataFusionResult { if let Some(plan) = self.last_plan.lock().as_ref() { return Ok(PyExecutionPlan::new(Arc::clone(plan))); } let plan = wait_for_future(py, self.df.as_ref().clone().create_physical_plan())??; Ok(plan.into()) } /// Repartition a `DataFrame` based on a logical partitioning scheme. fn repartition(&self, num: usize) -> PyDataFusionResult { let new_df = self .df .as_ref() .clone() .repartition(Partitioning::RoundRobinBatch(num))?; Ok(Self::new(new_df)) } /// Repartition a `DataFrame` based on a logical partitioning scheme. #[pyo3(signature = (*args, num))] fn repartition_by_hash(&self, args: Vec, num: usize) -> PyDataFusionResult { let expr = args.into_iter().map(|py_expr| py_expr.into()).collect(); let new_df = self .df .as_ref() .clone() .repartition(Partitioning::Hash(expr, num))?; Ok(Self::new(new_df)) } /// Calculate the union of two `DataFrame`s, preserving duplicate rows.The /// two `DataFrame`s must have exactly the same schema #[pyo3(signature = (py_df, distinct=false))] fn union(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { let new_df = if distinct { self.df .as_ref() .clone() .union_distinct(py_df.df.as_ref().clone())? } else { self.df.as_ref().clone().union(py_df.df.as_ref().clone())? }; Ok(Self::new(new_df)) } #[pyo3(signature = (columns, preserve_nulls=true, recursions=None))] fn unnest_columns( &self, columns: Vec, preserve_nulls: bool, recursions: Option>, ) -> PyDataFusionResult { let unnest_options = build_unnest_options(preserve_nulls, recursions); let cols = columns.iter().map(|s| s.as_ref()).collect::>(); let df = self .df .as_ref() .clone() .unnest_columns_with_options(&cols, unnest_options)?; Ok(Self::new(df)) } /// Calculate the intersection of two `DataFrame`s. The two `DataFrame`s must have exactly the same schema #[pyo3(signature = (py_df, distinct=false))] fn intersect(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { let base = self.df.as_ref().clone(); let other = py_df.df.as_ref().clone(); let new_df = if distinct { base.intersect_distinct(other)? } else { base.intersect(other)? }; Ok(Self::new(new_df)) } /// Calculate the exception of two `DataFrame`s. The two `DataFrame`s must have exactly the same schema #[pyo3(signature = (py_df, distinct=false))] fn except_all(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { let base = self.df.as_ref().clone(); let other = py_df.df.as_ref().clone(); let new_df = if distinct { base.except_distinct(other)? } else { base.except(other)? }; Ok(Self::new(new_df)) } /// Union two DataFrames matching columns by name #[pyo3(signature = (py_df, distinct=false))] fn union_by_name(&self, py_df: PyDataFrame, distinct: bool) -> PyDataFusionResult { let base = self.df.as_ref().clone(); let other = py_df.df.as_ref().clone(); let new_df = if distinct { base.union_by_name_distinct(other)? } else { base.union_by_name(other)? }; Ok(Self::new(new_df)) } /// Deduplicate rows based on specific columns, keeping the first row per group fn distinct_on( &self, on_expr: Vec, select_expr: Vec, sort_expr: Option>, ) -> PyDataFusionResult { let on_expr = on_expr.into_iter().map(|e| e.into()).collect(); let select_expr = select_expr.into_iter().map(|e| e.into()).collect(); let sort_expr = sort_expr.map(to_sort_expressions); let df = self .df .as_ref() .clone() .distinct_on(on_expr, select_expr, sort_expr)?; Ok(Self::new(df)) } /// Sort by column expressions with ascending order and nulls last fn sort_by(&self, exprs: Vec) -> PyDataFusionResult { let exprs = exprs.into_iter().map(|e| e.into()).collect(); let df = self.df.as_ref().clone().sort_by(exprs)?; Ok(Self::new(df)) } /// Return fully qualified column expressions for the given column names fn find_qualified_columns(&self, names: Vec) -> PyDataFusionResult> { let name_refs: Vec<&str> = names.iter().map(|s| s.as_str()).collect(); let qualified = self.df.find_qualified_columns(&name_refs)?; Ok(qualified .into_iter() .map(|q| Expr::Column(Column::from(q)).into()) .collect()) } /// Write a `DataFrame` to a CSV file. fn write_csv( &self, py: Python, path: &str, with_header: bool, write_options: Option, ) -> PyDataFusionResult<()> { let csv_options = CsvOptions { has_header: Some(with_header), ..Default::default() }; let write_options = write_options .map(DataFrameWriteOptions::from) .unwrap_or_default(); wait_for_future( py, self.df .as_ref() .clone() .write_csv(path, write_options, Some(csv_options)), )??; Ok(()) } /// Write a `DataFrame` to a Parquet file. #[pyo3(signature = ( path, compression="zstd", compression_level=None, write_options=None, ))] fn write_parquet( &self, path: &str, compression: &str, compression_level: Option, write_options: Option, py: Python, ) -> PyDataFusionResult<()> { fn verify_compression_level(cl: Option) -> Result { cl.ok_or(PyValueError::new_err("compression_level is not defined")) } let _validated = match compression.to_lowercase().as_str() { "snappy" => Compression::SNAPPY, "gzip" => Compression::GZIP( GzipLevel::try_new(compression_level.unwrap_or(6)) .map_err(|e| PyValueError::new_err(format!("{e}")))?, ), "brotli" => Compression::BROTLI( BrotliLevel::try_new(verify_compression_level(compression_level)?) .map_err(|e| PyValueError::new_err(format!("{e}")))?, ), "zstd" => Compression::ZSTD( ZstdLevel::try_new(verify_compression_level(compression_level)? as i32) .map_err(|e| PyValueError::new_err(format!("{e}")))?, ), "lzo" => Compression::LZO, "lz4" => Compression::LZ4, "lz4_raw" => Compression::LZ4_RAW, "uncompressed" => Compression::UNCOMPRESSED, _ => { return Err(PyDataFusionError::Common(format!( "Unrecognized compression type {compression}" ))); } }; let mut compression_string = compression.to_string(); if let Some(level) = compression_level { compression_string.push_str(&format!("({level})")); } let mut options = TableParquetOptions::default(); options.global.compression = Some(compression_string); let write_options = write_options .map(DataFrameWriteOptions::from) .unwrap_or_default(); wait_for_future( py, self.df .as_ref() .clone() .write_parquet(path, write_options, Option::from(options)), )??; Ok(()) } /// Write a `DataFrame` to a Parquet file, using advanced options. fn write_parquet_with_options( &self, path: &str, options: PyParquetWriterOptions, column_specific_options: HashMap, write_options: Option, py: Python, ) -> PyDataFusionResult<()> { let table_options = TableParquetOptions { global: options.options, column_specific_options: column_specific_options .into_iter() .map(|(k, v)| (k, v.options)) .collect(), ..Default::default() }; let write_options = write_options .map(DataFrameWriteOptions::from) .unwrap_or_default(); wait_for_future( py, self.df.as_ref().clone().write_parquet( path, write_options, Option::from(table_options), ), )??; Ok(()) } /// Executes a query and writes the results to a partitioned JSON file. fn write_json( &self, path: &str, py: Python, write_options: Option, ) -> PyDataFusionResult<()> { let write_options = write_options .map(DataFrameWriteOptions::from) .unwrap_or_default(); wait_for_future( py, self.df .as_ref() .clone() .write_json(path, write_options, None), )??; Ok(()) } fn write_table( &self, py: Python, table_name: &str, write_options: Option, ) -> PyDataFusionResult<()> { let write_options = write_options .map(DataFrameWriteOptions::from) .unwrap_or_default(); wait_for_future( py, self.df .as_ref() .clone() .write_table(table_name, write_options), )??; Ok(()) } /// Convert to Arrow Table /// Collect the batches and pass to Arrow Table fn to_arrow_table(&self, py: Python<'_>) -> PyResult> { let batches = self.collect(py)?.into_pyobject(py)?; // only use the DataFrame's schema if there are no batches, otherwise let the schema be // determined from the batches (avoids some inconsistencies with nullable columns) let args = if batches.len()? == 0 { let schema = self.schema().into_pyobject(py)?; PyTuple::new(py, &[batches, schema])? } else { PyTuple::new(py, &[batches])? }; // Instantiate pyarrow Table object and use its from_batches method let table_class = py.import("pyarrow")?.getattr("Table")?; let table: Py = table_class.call_method1("from_batches", args)?.into(); Ok(table) } #[pyo3(signature = (requested_schema=None))] fn __arrow_c_stream__<'py>( &'py self, py: Python<'py>, requested_schema: Option>, ) -> PyDataFusionResult> { let df = self.df.as_ref().clone(); let streams = spawn_future(py, async move { df.execute_stream_partitioned().await })?; let mut schema: Schema = self.df.schema().to_owned().as_arrow().clone(); let mut projection: Option = None; if let Some(schema_capsule) = requested_schema { let data: NonNull = schema_capsule .pointer_checked(Some(c"arrow_schema"))? .cast(); let schema_ptr = unsafe { data.as_ref() }; let desired_schema = Schema::try_from(schema_ptr)?; schema = project_schema(schema, desired_schema)?; projection = Some(Arc::new(schema.clone())); } let schema_ref = Arc::new(schema.clone()); let reader = PartitionedDataFrameStreamReader { streams, schema: schema_ref, projection, current: 0, }; let reader: Box = Box::new(reader); // Create the Arrow stream and wrap it in a PyCapsule. The default // destructor provided by PyO3 will drop the stream unless ownership is // transferred to PyArrow during import. let stream = FFI_ArrowArrayStream::new(reader); let name = CString::new(ARROW_ARRAY_STREAM_NAME.to_bytes()).unwrap(); let capsule = PyCapsule::new(py, stream, Some(name))?; Ok(capsule) } fn execute_stream(&self, py: Python) -> PyDataFusionResult { let (plan, task_ctx) = self.create_and_cache_plan(py)?; let stream = spawn_future(py, async move { df_execute_stream(plan, task_ctx) })?; Ok(PyRecordBatchStream::new(stream)) } fn execute_stream_partitioned(&self, py: Python) -> PyResult> { let (plan, task_ctx) = self.create_and_cache_plan(py)?; let streams = spawn_future( py, async move { df_execute_stream_partitioned(plan, task_ctx) }, )?; Ok(streams.into_iter().map(PyRecordBatchStream::new).collect()) } /// Convert to pandas dataframe with pyarrow /// Collect the batches, pass to Arrow Table & then convert to Pandas DataFrame fn to_pandas(&self, py: Python<'_>) -> PyResult> { let table = self.to_arrow_table(py)?; // See also: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pandas let result = table.call_method0(py, "to_pandas")?; Ok(result) } /// Convert to Python list using pyarrow /// Each list item represents one row encoded as dictionary fn to_pylist(&self, py: Python<'_>) -> PyResult> { let table = self.to_arrow_table(py)?; // See also: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pylist let result = table.call_method0(py, "to_pylist")?; Ok(result) } /// Convert to Python dictionary using pyarrow /// Each dictionary key is a column and the dictionary value represents the column values fn to_pydict(&self, py: Python) -> PyResult> { let table = self.to_arrow_table(py)?; // See also: https://arrow.apache.org/docs/python/generated/pyarrow.Table.html#pyarrow.Table.to_pydict let result = table.call_method0(py, "to_pydict")?; Ok(result) } /// Convert to polars dataframe with pyarrow /// Collect the batches, pass to Arrow Table & then convert to polars DataFrame fn to_polars(&self, py: Python<'_>) -> PyResult> { let table = self.to_arrow_table(py)?; let dataframe = py.import("polars")?.getattr("DataFrame")?; let args = PyTuple::new(py, &[table])?; let result: Py = dataframe.call1(args)?.into(); Ok(result) } // Executes this DataFrame to get the total number of rows. fn count(&self, py: Python) -> PyDataFusionResult { Ok(wait_for_future(py, self.df.as_ref().clone().count())??) } /// Fill null values with a specified value for specific columns #[pyo3(signature = (value, columns=None))] fn fill_null( &self, value: Py, columns: Option>, py: Python, ) -> PyDataFusionResult { let scalar_value: PyScalarValue = value.extract(py)?; let cols = match columns { Some(col_names) => col_names.iter().map(|c| c.to_string()).collect(), None => Vec::new(), // Empty vector means fill null for all columns }; let df = self.df.as_ref().clone().fill_null(scalar_value.0, cols)?; Ok(Self::new(df)) } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "InsertOp", module = "datafusion" )] pub enum PyInsertOp { APPEND, REPLACE, OVERWRITE, } impl From for InsertOp { fn from(value: PyInsertOp) -> Self { match value { PyInsertOp::APPEND => InsertOp::Append, PyInsertOp::REPLACE => InsertOp::Replace, PyInsertOp::OVERWRITE => InsertOp::Overwrite, } } } #[derive(Debug, Clone)] #[pyclass( from_py_object, frozen, name = "DataFrameWriteOptions", module = "datafusion" )] pub struct PyDataFrameWriteOptions { insert_operation: InsertOp, single_file_output: bool, partition_by: Vec, sort_by: Vec, } impl From for DataFrameWriteOptions { fn from(value: PyDataFrameWriteOptions) -> Self { DataFrameWriteOptions::new() .with_insert_operation(value.insert_operation) .with_single_file_output(value.single_file_output) .with_partition_by(value.partition_by) .with_sort_by(value.sort_by) } } #[pymethods] impl PyDataFrameWriteOptions { #[new] fn new( insert_operation: Option, single_file_output: bool, partition_by: Option>, sort_by: Option>, ) -> Self { let insert_operation = insert_operation.map(Into::into).unwrap_or(InsertOp::Append); let sort_by = sort_by .unwrap_or_default() .into_iter() .map(Into::into) .collect(); Self { insert_operation, single_file_output, partition_by: partition_by.unwrap_or_default(), sort_by, } } } fn build_unnest_options( preserve_nulls: bool, recursions: Option>, ) -> UnnestOptions { let mut opts = UnnestOptions::default().with_preserve_nulls(preserve_nulls); if let Some(recs) = recursions { opts.recursions = recs .into_iter() .map( |(input, output, depth)| datafusion::common::RecursionUnnestOption { input_column: datafusion::common::Column::from(input.as_str()), output_column: datafusion::common::Column::from(output.as_str()), depth, }, ) .collect(); } opts } /// Print DataFrame fn print_dataframe(py: Python, df: DataFrame) -> PyDataFusionResult<()> { // Get string representation of record batches let batches = wait_for_future(py, df.collect())??; let result = if batches.is_empty() { "DataFrame has no rows".to_string() } else { match pretty::pretty_format_batches(&batches) { Ok(batch) => format!("DataFrame()\n{batch}"), Err(err) => format!("Error: {:?}", err.to_string()), } }; // Import the Python 'builtins' module to access the print function // Note that println! does not print to the Python debug console and is not visible in notebooks for instance let print = py.import("builtins")?.getattr("print")?; print.call1((result,))?; Ok(()) } fn project_schema(from_schema: Schema, to_schema: Schema) -> Result { let merged_schema = Schema::try_merge(vec![from_schema, to_schema.clone()])?; let project_indices: Vec = to_schema .fields .iter() .map(|field| field.name()) .filter_map(|field_name| merged_schema.index_of(field_name).ok()) .collect(); merged_schema.project(&project_indices) } // NOTE: `arrow::compute::cast` in combination with `RecordBatch::try_select` or // DataFusion's `schema::cast_record_batch` do not fully cover the required // transformations here. They will not create missing columns and may insert // nulls for non-nullable fields without erroring. To maintain current behavior // we perform the casting and null checks manually. fn record_batch_into_schema( record_batch: RecordBatch, schema: &Schema, ) -> Result { let schema = Arc::new(schema.clone()); let base_schema = record_batch.schema(); if base_schema.fields().is_empty() { // Nothing to project return Ok(RecordBatch::new_empty(schema)); } let array_size = record_batch.column(0).len(); let mut data_arrays = Vec::with_capacity(schema.fields().len()); for field in schema.fields() { let desired_data_type = field.data_type(); if let Some(original_data) = record_batch.column_by_name(field.name()) { let original_data_type = original_data.data_type(); if can_cast_types(original_data_type, desired_data_type) { data_arrays.push(arrow::compute::kernels::cast( original_data, desired_data_type, )?); } else if field.is_nullable() { data_arrays.push(new_null_array(desired_data_type, array_size)); } else { return Err(ArrowError::CastError(format!( "Attempting to cast to non-nullable and non-castable field {} during schema projection.", field.name() ))); } } else { if !field.is_nullable() { return Err(ArrowError::CastError(format!( "Attempting to set null to non-nullable field {} during schema projection.", field.name() ))); } data_arrays.push(new_null_array(desired_data_type, array_size)); } } RecordBatch::try_new(schema, data_arrays) } /// This is a helper function to return the first non-empty record batch from executing a DataFrame. /// It additionally returns a bool, which indicates if there are more record batches available. /// We do this so we can determine if we should indicate to the user that the data has been /// truncated. This collects until we have archived both of these two conditions /// /// - We have collected our minimum number of rows /// - We have reached our limit, either data size or maximum number of rows /// /// Otherwise it will return when the stream has exhausted. If you want a specific number of /// rows, set min_rows == max_rows. async fn collect_record_batches_to_display( df: DataFrame, config: FormatterConfig, ) -> Result<(Vec, bool), DataFusionError> { let FormatterConfig { max_bytes, min_rows, max_rows, } = config; let partitioned_stream = df.execute_stream_partitioned().await?; let mut stream = futures::stream::iter(partitioned_stream).flatten(); let mut size_estimate_so_far = 0; let mut rows_so_far = 0; let mut record_batches = Vec::default(); let mut has_more = false; // Collect rows until we hit a limit (memory or max_rows) OR reach the guaranteed minimum. // The minimum rows constraint overrides both memory and row limits to ensure a baseline // of data is always displayed, even if it temporarily exceeds those limits. // This provides better UX by guaranteeing users see at least min_rows rows. while (size_estimate_so_far < max_bytes && rows_so_far < max_rows) || rows_so_far < min_rows { let mut rb = match stream.next().await { None => { break; } Some(Ok(r)) => r, Some(Err(e)) => return Err(e), }; let mut rows_in_rb = rb.num_rows(); if rows_in_rb > 0 { size_estimate_so_far += rb.get_array_memory_size(); // When memory limit is exceeded, scale back row count proportionally to stay within budget if size_estimate_so_far > max_bytes { let ratio = max_bytes as f32 / size_estimate_so_far as f32; let total_rows = rows_in_rb + rows_so_far; // Calculate reduced rows maintaining the memory/data proportion let mut reduced_row_num = (total_rows as f32 * ratio).round() as usize; // Ensure we always respect the minimum rows guarantee if reduced_row_num < min_rows { reduced_row_num = min_rows.min(total_rows); } let limited_rows_this_rb = reduced_row_num - rows_so_far; if limited_rows_this_rb < rows_in_rb { rows_in_rb = limited_rows_this_rb; rb = rb.slice(0, limited_rows_this_rb); has_more = true; } } if rows_in_rb + rows_so_far > max_rows { rb = rb.slice(0, max_rows - rows_so_far); has_more = true; } rows_so_far += rb.num_rows(); record_batches.push(rb); } } if record_batches.is_empty() { return Ok((Vec::default(), false)); } if !has_more { // Data was not already truncated, so check to see if more record batches remain has_more = match stream.try_next().await { Ok(None) => false, // reached end Ok(Some(_)) => true, Err(_) => false, // Stream disconnected }; } Ok((record_batches, has_more)) } apache-datafusion-python-54.0.0/crates/core/src/dataset.rs000066400000000000000000000110751521727353300235020ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use async_trait::async_trait; use datafusion::arrow::datatypes::SchemaRef; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::catalog::Session; use datafusion::datasource::{TableProvider, TableType}; use datafusion::error::{DataFusionError, Result as DFResult}; use datafusion::logical_expr::{Expr, TableProviderFilterPushDown}; use datafusion::physical_plan::ExecutionPlan; use pyo3::exceptions::PyValueError; /// Implements a Datafusion TableProvider that delegates to a PyArrow Dataset /// This allows us to use PyArrow Datasets as Datafusion tables while pushing down projections and filters use pyo3::prelude::*; use pyo3::types::PyType; use crate::dataset_exec::DatasetExec; use crate::pyarrow_filter_expression::PyArrowFilterExpression; // Wraps a pyarrow.dataset.Dataset class and implements a Datafusion TableProvider around it #[derive(Debug)] pub(crate) struct Dataset { dataset: Py, } impl Dataset { // Creates a Python PyArrow.Dataset pub fn new(dataset: &Bound<'_, PyAny>, py: Python) -> PyResult { // Ensure that we were passed an instance of pyarrow.dataset.Dataset let ds = PyModule::import(py, "pyarrow.dataset")?; let ds_attr = ds.getattr("Dataset")?; let ds_type = ds_attr.cast::()?; if dataset.is_instance(ds_type)? { Ok(Dataset { dataset: dataset.clone().unbind(), }) } else { Err(PyValueError::new_err( "dataset argument must be a pyarrow.dataset.Dataset object", )) } } } #[async_trait] impl TableProvider for Dataset { /// Get a reference to the schema for this table fn schema(&self) -> SchemaRef { Python::attach(|py| { let dataset = self.dataset.bind(py); // This can panic but since we checked that self.dataset is a pyarrow.dataset.Dataset it should never Arc::new( dataset .getattr("schema") .unwrap() .extract::>() .unwrap() .0, ) }) } /// Get the type of this table for metadata/catalog purposes. fn table_type(&self) -> TableType { TableType::Base } /// Create an ExecutionPlan that will scan the table. /// The table provider will be usually responsible of grouping /// the source data into partitions that can be efficiently /// parallelized or distributed. async fn scan( &self, _ctx: &dyn Session, projection: Option<&Vec>, filters: &[Expr], // limit can be used to reduce the amount scanned // from the datasource as a performance optimization. // If set, it contains the amount of rows needed by the `LogicalPlan`, // The datasource should return *at least* this number of rows if available. _limit: Option, ) -> DFResult> { Python::attach(|py| { let plan: Arc = Arc::new( DatasetExec::new(py, self.dataset.bind(py), projection.cloned(), filters) .map_err(|err| DataFusionError::External(Box::new(err)))?, ); Ok(plan) }) } /// Tests whether the table provider can make use of a filter expression /// to optimise data retrieval. fn supports_filters_pushdown( &self, filter: &[&Expr], ) -> DFResult> { filter .iter() .map(|&f| match PyArrowFilterExpression::try_from(f) { Ok(_) => Ok(TableProviderFilterPushDown::Exact), _ => Ok(TableProviderFilterPushDown::Unsupported), }) .collect() } } apache-datafusion-python-54.0.0/crates/core/src/dataset_exec.rs000066400000000000000000000254641521727353300245150ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::arrow::datatypes::SchemaRef; use datafusion::arrow::error::{ArrowError, Result as ArrowResult}; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::arrow::record_batch::RecordBatch; use datafusion::error::{DataFusionError as InnerDataFusionError, Result as DFResult}; use datafusion::execution::context::TaskContext; use datafusion::logical_expr::Expr; use datafusion::logical_expr::utils::conjunction; use datafusion::physical_expr::{EquivalenceProperties, LexOrdering}; use datafusion::physical_plan::execution_plan::{Boundedness, EmissionType}; use datafusion::physical_plan::stream::RecordBatchStreamAdapter; use datafusion::physical_plan::{ DisplayAs, DisplayFormatType, ExecutionPlan, ExecutionPlanProperties, Partitioning, PlanProperties, SendableRecordBatchStream, Statistics, }; use futures::{TryStreamExt, stream}; /// Implements a Datafusion physical ExecutionPlan that delegates to a PyArrow Dataset /// This actually performs the projection, filtering and scanning of a Dataset use pyo3::prelude::*; use pyo3::types::{PyDict, PyIterator, PyList}; use crate::errors::PyDataFusionResult; use crate::pyarrow_filter_expression::PyArrowFilterExpression; struct PyArrowBatchesAdapter { batches: Py, } impl Iterator for PyArrowBatchesAdapter { type Item = ArrowResult; fn next(&mut self) -> Option { Python::attach(|py| { let mut batches = self.batches.clone_ref(py).into_bound(py); Some( batches .next()? .and_then(|batch| Ok(batch.extract::>()?.0)) .map_err(|err| ArrowError::ExternalError(Box::new(err))), ) }) } } // Wraps a pyarrow.dataset.Dataset class and implements a Datafusion ExecutionPlan around it #[derive(Debug)] pub(crate) struct DatasetExec { dataset: Py, schema: SchemaRef, fragments: Py, columns: Option>, filter_expr: Option>, projected_statistics: Statistics, plan_properties: Arc, } impl DatasetExec { pub fn new( py: Python, dataset: &Bound<'_, PyAny>, projection: Option>, filters: &[Expr], ) -> PyDataFusionResult { let columns: Option>> = projection.map(|p| { p.iter() .map(|index| { let name: String = dataset .getattr("schema")? .call_method1("field", (*index,))? .getattr("name")? .extract()?; Ok(name) }) .collect() }); let columns: Option> = columns.transpose()?; let filter_expr: Option> = conjunction(filters.to_owned()) .map(|filters| { PyArrowFilterExpression::try_from(&filters) .map(|filter_expr| filter_expr.inner().clone_ref(py)) }) .transpose()?; let kwargs = PyDict::new(py); kwargs.set_item("columns", columns.clone())?; kwargs.set_item( "filter", filter_expr.as_ref().map(|expr| expr.clone_ref(py)), )?; let scanner = dataset.call_method("scanner", (), Some(&kwargs))?; let schema: SchemaRef = Arc::new( scanner .getattr("projected_schema")? .extract::>()? .0, ); let builtins = Python::import(py, "builtins")?; let pylist = builtins.getattr("list")?; // Get the fragments or partitions of the dataset let fragments_iterator: Bound<'_, PyAny> = dataset.call_method1( "get_fragments", (filter_expr.as_ref().map(|expr| expr.clone_ref(py)),), )?; let fragments_iter = pylist.call1((fragments_iterator,))?; let fragments = fragments_iter.cast::().map_err(PyErr::from)?; let projected_statistics = Statistics::new_unknown(&schema); let plan_properties = Arc::new(PlanProperties::new( EquivalenceProperties::new(schema.clone()), Partitioning::UnknownPartitioning(fragments.len()), EmissionType::Final, Boundedness::Bounded, )); Ok(DatasetExec { dataset: dataset.clone().unbind(), schema, fragments: fragments.clone().unbind(), columns, filter_expr, projected_statistics, plan_properties, }) } } impl ExecutionPlan for DatasetExec { fn name(&self) -> &str { // [ExecutionPlan::name] docs recommends forwarding to `static_name` Self::static_name() } /// Get the schema for this execution plan fn schema(&self) -> SchemaRef { self.schema.clone() } fn children(&self) -> Vec<&Arc> { // this is a leaf node and has no children vec![] } fn with_new_children( self: Arc, _: Vec>, ) -> DFResult> { Ok(self) } fn execute( &self, partition: usize, context: Arc, ) -> DFResult { let batch_size = context.session_config().batch_size(); Python::attach(|py| { let dataset = self.dataset.bind(py); let fragments = self.fragments.bind(py); let fragment = fragments .get_item(partition) .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; // We need to pass the dataset schema to unify the fragment and dataset schema per PyArrow docs let dataset_schema = dataset .getattr("schema") .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; let kwargs = PyDict::new(py); kwargs .set_item("columns", self.columns.clone()) .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; kwargs .set_item( "filter", self.filter_expr.as_ref().map(|expr| expr.clone_ref(py)), ) .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; kwargs .set_item("batch_size", batch_size) .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; let scanner = fragment .call_method("scanner", (dataset_schema,), Some(&kwargs)) .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; let schema: SchemaRef = Arc::new( scanner .getattr("projected_schema") .and_then(|schema| Ok(schema.extract::>()?.0)) .map_err(|err| InnerDataFusionError::External(Box::new(err)))?, ); let record_batches: Bound<'_, PyIterator> = scanner .call_method0("to_batches") .map_err(|err| InnerDataFusionError::External(Box::new(err)))? .try_iter() .map_err(|err| InnerDataFusionError::External(Box::new(err)))?; let record_batches = PyArrowBatchesAdapter { batches: record_batches.into(), }; let record_batch_stream = stream::iter(record_batches); let record_batch_stream: SendableRecordBatchStream = Box::pin( RecordBatchStreamAdapter::new(schema, record_batch_stream.map_err(|e| e.into())), ); Ok(record_batch_stream) }) } fn partition_statistics(&self, _partition: Option) -> DFResult> { Ok(Arc::new(self.projected_statistics.clone())) } fn properties(&self) -> &Arc { &self.plan_properties } } impl ExecutionPlanProperties for DatasetExec { /// Get the output partitioning of this plan fn output_partitioning(&self) -> &Partitioning { self.plan_properties.output_partitioning() } fn output_ordering(&self) -> Option<&LexOrdering> { None } fn boundedness(&self) -> Boundedness { self.plan_properties.boundedness } fn pipeline_behavior(&self) -> EmissionType { self.plan_properties.emission_type } fn equivalence_properties(&self) -> &datafusion::physical_expr::EquivalenceProperties { &self.plan_properties.eq_properties } } impl DisplayAs for DatasetExec { fn fmt_as(&self, t: DisplayFormatType, f: &mut std::fmt::Formatter) -> std::fmt::Result { Python::attach(|py| { let number_of_fragments = self.fragments.bind(py).len(); match t { DisplayFormatType::Default | DisplayFormatType::Verbose | DisplayFormatType::TreeRender => { let projected_columns: Vec = self .schema .fields() .iter() .map(|x| x.name().to_owned()) .collect(); if let Some(filter_expr) = &self.filter_expr { let filter_expr = filter_expr.bind(py).str().or(Err(std::fmt::Error))?; write!( f, "DatasetExec: number_of_fragments={}, filter_expr={}, projection=[{}]", number_of_fragments, filter_expr, projected_columns.join(", "), ) } else { write!( f, "DatasetExec: number_of_fragments={}, projection=[{}]", number_of_fragments, projected_columns.join(", "), ) } } } }) } } apache-datafusion-python-54.0.0/crates/core/src/errors.rs000066400000000000000000000015151521727353300233670ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. pub use datafusion_python_util::errors::*; apache-datafusion-python-54.0.0/crates/core/src/expr.rs000066400000000000000000001106671521727353300230420ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::convert::{From, Into}; use std::sync::Arc; use datafusion::arrow::datatypes::{DataType, Field}; use datafusion::arrow::pyarrow::PyArrowType; use datafusion::functions::core::expr_ext::FieldAccessor; use datafusion::logical_expr::expr::{ AggregateFunction, AggregateFunctionParams, FieldMetadata, HigherOrderFunction, InList, InSubquery, Lambda, ScalarFunction, SetComparison, WindowFunction, }; use datafusion::logical_expr::utils::exprlist_to_fields; use datafusion::logical_expr::{ Between, BinaryExpr, Case, Cast, Expr, ExprFuncBuilder, ExprFunctionExt, Like, LogicalPlan, Operator, TryCast, WindowFunctionDefinition, col, lit, lit_with_metadata, }; use datafusion_proto::logical_plan::{from_proto, to_proto}; use prost::Message; use pyo3::IntoPyObjectExt; use pyo3::basic::CompareOp; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use pyo3::types::PyBytes; use window::PyWindowFrame; use self::alias::PyAlias; use self::bool_expr::{ PyIsFalse, PyIsNotFalse, PyIsNotNull, PyIsNotTrue, PyIsNotUnknown, PyIsNull, PyIsTrue, PyIsUnknown, PyNegative, PyNot, }; use self::like::{PyILike, PyLike, PySimilarTo}; use self::scalar_variable::PyScalarVariable; use crate::codec::PythonLogicalCodec; use crate::common::data_type::{DataTypeMap, NullTreatment, PyScalarValue, RexType}; use crate::context::PySessionContext; use crate::errors::{PyDataFusionResult, py_runtime_err, py_type_err, py_unsupported_variant_err}; use crate::expr::aggregate_expr::PyAggregateFunction; use crate::expr::binary_expr::PyBinaryExpr; use crate::expr::column::PyColumn; use crate::expr::literal::PyLiteral; use crate::functions::add_builder_fns_to_window; use crate::pyarrow_util::scalar_to_pyarrow; use crate::sql::logical::PyLogicalPlan; pub mod aggregate; pub mod aggregate_expr; pub mod alias; pub mod analyze; pub mod between; pub mod binary_expr; pub mod bool_expr; pub mod case; pub mod cast; pub mod column; pub mod conditional_expr; pub mod copy_to; pub mod create_catalog; pub mod create_catalog_schema; pub mod create_external_table; pub mod create_function; pub mod create_index; pub mod create_memory_table; pub mod create_view; pub mod describe_table; pub mod distinct; pub mod dml; pub mod drop_catalog_schema; pub mod drop_function; pub mod drop_table; pub mod drop_view; pub mod empty_relation; pub mod exists; pub mod explain; pub mod extension; pub mod filter; pub mod grouping_set; pub mod higher_order_function; pub mod in_list; pub mod in_subquery; pub mod join; pub mod lambda; pub mod lambda_variable; pub mod like; pub mod limit; pub mod literal; pub mod logical_node; pub mod placeholder; pub mod projection; pub mod recursive_query; pub mod repartition; pub mod scalar_subquery; pub mod scalar_variable; pub mod set_comparison; pub mod signature; pub mod sort; pub mod sort_expr; pub mod statement; pub mod subquery; pub mod subquery_alias; pub mod table_scan; pub mod union; pub mod unnest; pub mod unnest_expr; pub mod values; pub mod window; use sort_expr::{PySortExpr, to_sort_expressions}; /// A PyExpr that can be used on a DataFrame #[pyclass( from_py_object, frozen, name = "RawExpr", module = "datafusion.expr", subclass )] #[derive(Debug, Clone)] pub struct PyExpr { pub expr: Expr, } impl From for Expr { fn from(expr: PyExpr) -> Expr { expr.expr } } impl From for PyExpr { fn from(expr: Expr) -> PyExpr { PyExpr { expr } } } /// Convert a list of DataFusion Expr to PyExpr pub fn py_expr_list(expr: &[Expr]) -> PyResult> { Ok(expr.iter().map(|e| PyExpr::from(e.clone())).collect()) } #[pymethods] impl PyExpr { /// Return the specific expression fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { Python::attach(|_| match &self.expr { Expr::Alias(alias) => Ok(PyAlias::from(alias.clone()).into_bound_py_any(py)?), Expr::Column(col) => Ok(PyColumn::from(col.clone()).into_bound_py_any(py)?), Expr::ScalarVariable(field, variables) => { Ok(PyScalarVariable::new(field, variables).into_bound_py_any(py)?) } Expr::Like(value) => Ok(PyLike::from(value.clone()).into_bound_py_any(py)?), Expr::Literal(value, metadata) => Ok(PyLiteral::new_with_metadata( value.clone(), metadata.clone(), ) .into_bound_py_any(py)?), Expr::BinaryExpr(expr) => Ok(PyBinaryExpr::from(expr.clone()).into_bound_py_any(py)?), Expr::Not(expr) => Ok(PyNot::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsNotNull(expr) => Ok(PyIsNotNull::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsNull(expr) => Ok(PyIsNull::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsTrue(expr) => Ok(PyIsTrue::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsFalse(expr) => Ok(PyIsFalse::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsUnknown(expr) => Ok(PyIsUnknown::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsNotTrue(expr) => Ok(PyIsNotTrue::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsNotFalse(expr) => Ok(PyIsNotFalse::new(*expr.clone()).into_bound_py_any(py)?), Expr::IsNotUnknown(expr) => { Ok(PyIsNotUnknown::new(*expr.clone()).into_bound_py_any(py)?) } Expr::Negative(expr) => Ok(PyNegative::new(*expr.clone()).into_bound_py_any(py)?), Expr::AggregateFunction(expr) => { Ok(PyAggregateFunction::from(expr.clone()).into_bound_py_any(py)?) } Expr::SimilarTo(value) => Ok(PySimilarTo::from(value.clone()).into_bound_py_any(py)?), Expr::Between(value) => { Ok(between::PyBetween::from(value.clone()).into_bound_py_any(py)?) } Expr::Case(value) => Ok(case::PyCase::from(value.clone()).into_bound_py_any(py)?), Expr::Cast(value) => Ok(cast::PyCast::from(value.clone()).into_bound_py_any(py)?), Expr::TryCast(value) => Ok(cast::PyTryCast::from(value.clone()).into_bound_py_any(py)?), Expr::ScalarFunction(value) => Err(py_unsupported_variant_err(format!( "Converting Expr::ScalarFunction to a Python object is not implemented: {value:?}" ))), Expr::WindowFunction(value) => Err(py_unsupported_variant_err(format!( "Converting Expr::WindowFunction to a Python object is not implemented: {value:?}" ))), Expr::InList(value) => { Ok(in_list::PyInList::from(value.clone()).into_bound_py_any(py)?) } Expr::Exists(value) => Ok(exists::PyExists::from(value.clone()).into_bound_py_any(py)?), Expr::InSubquery(value) => { Ok(in_subquery::PyInSubquery::from(value.clone()).into_bound_py_any(py)?) } Expr::ScalarSubquery(value) => { Ok(scalar_subquery::PyScalarSubquery::from(value.clone()).into_bound_py_any(py)?) } #[allow(deprecated)] Expr::Wildcard { qualifier, options } => Err(py_unsupported_variant_err(format!( "Converting Expr::Wildcard to a Python object is not implemented : {qualifier:?} {options:?}" ))), Expr::GroupingSet(value) => { Ok(grouping_set::PyGroupingSet::from(value.clone()).into_bound_py_any(py)?) } Expr::Placeholder(value) => { Ok(placeholder::PyPlaceholder::from(value.clone()).into_bound_py_any(py)?) } Expr::OuterReferenceColumn(data_type, column) => { Err(py_unsupported_variant_err(format!( "Converting Expr::OuterReferenceColumn to a Python object is not implemented: {data_type:?} - {column:?}" ))) } Expr::Unnest(value) => { Ok(unnest_expr::PyUnnestExpr::from(value.clone()).into_bound_py_any(py)?) } Expr::SetComparison(value) => { Ok(set_comparison::PySetComparison::from(value.clone()).into_bound_py_any(py)?) } Expr::HigherOrderFunction(value) => Ok( higher_order_function::PyHigherOrderFunction::from(value.clone()) .into_bound_py_any(py)?, ), Expr::Lambda(value) => Ok(lambda::PyLambda::from(value.clone()).into_bound_py_any(py)?), Expr::LambdaVariable(value) => { Ok(lambda_variable::PyLambdaVariable::from(value.clone()).into_bound_py_any(py)?) } }) } /// Returns the name of this expression as it should appear in a schema. This name /// will not include any CAST expressions. fn schema_name(&self) -> PyResult { Ok(format!("{}", self.expr.schema_name())) } /// Returns a full and complete string representation of this expression. fn canonical_name(&self) -> PyResult { Ok(format!("{}", self.expr)) } /// Returns the name of the Expr variant. /// Ex: 'IsNotNull', 'Literal', 'BinaryExpr', etc fn variant_name(&self) -> PyResult<&str> { Ok(self.expr.variant_name()) } fn __richcmp__(&self, other: PyExpr, op: CompareOp) -> PyExpr { let expr = match op { CompareOp::Lt => self.expr.clone().lt(other.expr), CompareOp::Le => self.expr.clone().lt_eq(other.expr), CompareOp::Eq => self.expr.clone().eq(other.expr), CompareOp::Ne => self.expr.clone().not_eq(other.expr), CompareOp::Gt => self.expr.clone().gt(other.expr), CompareOp::Ge => self.expr.clone().gt_eq(other.expr), }; expr.into() } fn __repr__(&self) -> PyResult { Ok(format!("Expr({})", self.expr)) } fn __add__(&self, rhs: PyExpr) -> PyResult { Ok((self.expr.clone() + rhs.expr).into()) } fn __sub__(&self, rhs: PyExpr) -> PyResult { Ok((self.expr.clone() - rhs.expr).into()) } fn __truediv__(&self, rhs: PyExpr) -> PyResult { Ok((self.expr.clone() / rhs.expr).into()) } fn __mul__(&self, rhs: PyExpr) -> PyResult { Ok((self.expr.clone() * rhs.expr).into()) } fn __mod__(&self, rhs: PyExpr) -> PyResult { let expr = self.expr.clone() % rhs.expr; Ok(expr.into()) } fn __and__(&self, rhs: PyExpr) -> PyResult { Ok(self.expr.clone().and(rhs.expr).into()) } fn __or__(&self, rhs: PyExpr) -> PyResult { Ok(self.expr.clone().or(rhs.expr).into()) } fn __invert__(&self) -> PyResult { let expr = !self.expr.clone(); Ok(expr.into()) } fn __getitem__(&self, key: &str) -> PyResult { Ok(self.expr.clone().field(key).into()) } #[staticmethod] pub fn literal(value: PyScalarValue) -> PyExpr { lit(value.0).into() } #[staticmethod] pub fn literal_with_metadata( value: PyScalarValue, metadata: HashMap, ) -> PyExpr { let metadata = FieldMetadata::new(metadata.into_iter().collect()); lit_with_metadata(value.0, Some(metadata)).into() } #[staticmethod] pub fn column(value: &str) -> PyExpr { col(value).into() } /// assign a name to the PyExpr #[pyo3(signature = (name, metadata=None))] pub fn alias(&self, name: &str, metadata: Option>) -> PyExpr { let metadata = metadata.map(|m| FieldMetadata::new(m.into_iter().collect())); self.expr.clone().alias_with_metadata(name, metadata).into() } /// Create a sort PyExpr from an existing PyExpr. #[pyo3(signature = (ascending=true, nulls_first=true))] pub fn sort(&self, ascending: bool, nulls_first: bool) -> PySortExpr { self.expr.clone().sort(ascending, nulls_first).into() } pub fn is_null(&self) -> PyExpr { self.expr.clone().is_null().into() } pub fn is_not_null(&self) -> PyExpr { self.expr.clone().is_not_null().into() } pub fn cast(&self, to: PyArrowType) -> PyExpr { // self.expr.cast_to() requires DFSchema to validate that the cast // is supported, omit that for now let expr = Expr::Cast(Cast::new(Box::new(self.expr.clone()), to.0)); expr.into() } pub fn try_cast(&self, to: PyArrowType) -> PyExpr { let expr = Expr::TryCast(TryCast::new(Box::new(self.expr.clone()), to.0)); expr.into() } #[pyo3(signature = (low, high, negated=false))] pub fn between(&self, low: PyExpr, high: PyExpr, negated: bool) -> PyExpr { let expr = Expr::Between(Between::new( Box::new(self.expr.clone()), negated, Box::new(low.into()), Box::new(high.into()), )); expr.into() } /// A Rex (Row Expression) specifies a single row of data. That specification /// could include user defined functions or types. RexType identifies the row /// as one of the possible valid `RexTypes`. pub fn rex_type(&self) -> PyResult { Ok(match self.expr { Expr::Alias(..) => RexType::Alias, Expr::Column(..) => RexType::Reference, Expr::ScalarVariable(..) | Expr::Literal(..) => RexType::Literal, Expr::BinaryExpr { .. } | Expr::Not(..) | Expr::IsNotNull(..) | Expr::Negative(..) | Expr::IsNull(..) | Expr::Like { .. } | Expr::SimilarTo { .. } | Expr::Between { .. } | Expr::Case { .. } | Expr::Cast { .. } | Expr::TryCast { .. } | Expr::ScalarFunction { .. } | Expr::AggregateFunction { .. } | Expr::WindowFunction { .. } | Expr::InList { .. } | Expr::Exists { .. } | Expr::InSubquery { .. } | Expr::GroupingSet(..) | Expr::IsTrue(..) | Expr::IsFalse(..) | Expr::IsUnknown(_) | Expr::IsNotTrue(..) | Expr::IsNotFalse(..) | Expr::Placeholder { .. } | Expr::OuterReferenceColumn(_, _) | Expr::Unnest(_) | Expr::IsNotUnknown(_) | Expr::SetComparison(_) | Expr::HigherOrderFunction(..) | Expr::Lambda(..) => RexType::Call, Expr::LambdaVariable(..) => RexType::Reference, Expr::ScalarSubquery(..) => RexType::ScalarSubquery, #[allow(deprecated)] Expr::Wildcard { .. } => { return Err(py_unsupported_variant_err("Expr::Wildcard is unsupported")); } }) } /// Given the current `Expr` return the DataTypeMap which represents the /// PythonType, Arrow DataType, and SqlType Enum which represents pub fn types(&self) -> PyResult { Self::_types(&self.expr) } /// Extracts the Expr value into a Py that can be shared with Python pub fn python_value<'py>(&self, py: Python<'py>) -> PyResult> { match &self.expr { Expr::Literal(scalar_value, _) => scalar_to_pyarrow(scalar_value, py), _ => Err(py_type_err(format!( "Non Expr::Literal encountered in types: {:?}", &self.expr ))), } } /// Row expressions, Rex(s), operate on the concept of operands. Different variants of Expressions, Expr(s), /// store those operands in different datastructures. This function examines the Expr variant and returns /// the operands to the calling logic as a Vec of PyExpr instances. pub fn rex_call_operands(&self) -> PyResult> { match &self.expr { // Expr variants that are themselves the operand to return Expr::Column(..) | Expr::ScalarVariable(..) | Expr::Literal(..) | Expr::LambdaVariable(..) => Ok(vec![PyExpr::from(self.expr.clone())]), Expr::Alias(alias) => Ok(vec![PyExpr::from(*alias.expr.clone())]), // Expr(s) that house the Expr instance to return in their bounded params Expr::Not(expr) | Expr::IsNull(expr) | Expr::IsNotNull(expr) | Expr::IsTrue(expr) | Expr::IsFalse(expr) | Expr::IsUnknown(expr) | Expr::IsNotTrue(expr) | Expr::IsNotFalse(expr) | Expr::IsNotUnknown(expr) | Expr::Negative(expr) | Expr::Cast(Cast { expr, .. }) | Expr::TryCast(TryCast { expr, .. }) | Expr::InSubquery(InSubquery { expr, .. }) | Expr::SetComparison(SetComparison { expr, .. }) => { Ok(vec![PyExpr::from(*expr.clone())]) } // Expr variants containing a collection of Expr(s) for operands Expr::AggregateFunction(AggregateFunction { params: AggregateFunctionParams { args, .. }, .. }) | Expr::ScalarFunction(ScalarFunction { args, .. }) | Expr::HigherOrderFunction(HigherOrderFunction { args, .. }) => { Ok(args.iter().map(|arg| PyExpr::from(arg.clone())).collect()) } Expr::WindowFunction(boxed_window_fn) => { let args = &boxed_window_fn.params.args; Ok(args.iter().map(|arg| PyExpr::from(arg.clone())).collect()) } Expr::Lambda(Lambda { body, .. }) => Ok(vec![PyExpr::from(*body.clone())]), // Expr(s) that require more specific processing Expr::Case(Case { expr, when_then_expr, else_expr, }) => { let mut operands: Vec = Vec::new(); if let Some(e) = expr { for (when, then) in when_then_expr { operands.push(PyExpr::from(Expr::BinaryExpr(BinaryExpr::new( Box::new(*e.clone()), Operator::Eq, Box::new(*when.clone()), )))); operands.push(PyExpr::from(*then.clone())); } } else { for (when, then) in when_then_expr { operands.push(PyExpr::from(*when.clone())); operands.push(PyExpr::from(*then.clone())); } }; if let Some(e) = else_expr { operands.push(PyExpr::from(*e.clone())); }; Ok(operands) } Expr::InList(InList { expr, list, .. }) => { let mut operands: Vec = vec![PyExpr::from(*expr.clone())]; for list_elem in list { operands.push(PyExpr::from(list_elem.clone())); } Ok(operands) } Expr::BinaryExpr(BinaryExpr { left, right, .. }) => Ok(vec![ PyExpr::from(*left.clone()), PyExpr::from(*right.clone()), ]), Expr::Like(Like { expr, pattern, .. }) => Ok(vec![ PyExpr::from(*expr.clone()), PyExpr::from(*pattern.clone()), ]), Expr::SimilarTo(Like { expr, pattern, .. }) => Ok(vec![ PyExpr::from(*expr.clone()), PyExpr::from(*pattern.clone()), ]), Expr::Between(Between { expr, negated: _, low, high, }) => Ok(vec![ PyExpr::from(*expr.clone()), PyExpr::from(*low.clone()), PyExpr::from(*high.clone()), ]), // Currently un-support/implemented Expr types for Rex Call operations Expr::GroupingSet(..) | Expr::Unnest(_) | Expr::OuterReferenceColumn(_, _) | Expr::ScalarSubquery(..) | Expr::Placeholder { .. } | Expr::Exists { .. } => Err(py_runtime_err(format!( "Unimplemented Expr type: {}", self.expr ))), #[allow(deprecated)] Expr::Wildcard { .. } => { Err(py_unsupported_variant_err("Expr::Wildcard is unsupported")) } } } /// Extracts the operator associated with a RexType::Call pub fn rex_call_operator(&self) -> PyResult { Ok(match &self.expr { Expr::BinaryExpr(BinaryExpr { left: _, op, right: _, }) => format!("{op}"), Expr::ScalarFunction(ScalarFunction { func, args: _ }) => func.name().to_string(), Expr::HigherOrderFunction(HigherOrderFunction { func, args: _ }) => { func.name().to_string() } Expr::Lambda(..) => "lambda".to_string(), Expr::Cast { .. } => "cast".to_string(), Expr::Between { .. } => "between".to_string(), Expr::Case { .. } => "case".to_string(), Expr::IsNull(..) => "is null".to_string(), Expr::IsNotNull(..) => "is not null".to_string(), Expr::IsTrue(_) => "is true".to_string(), Expr::IsFalse(_) => "is false".to_string(), Expr::IsUnknown(_) => "is unknown".to_string(), Expr::IsNotTrue(_) => "is not true".to_string(), Expr::IsNotFalse(_) => "is not false".to_string(), Expr::IsNotUnknown(_) => "is not unknown".to_string(), Expr::InList { .. } => "in list".to_string(), Expr::Negative(..) => "negative".to_string(), Expr::Not(..) => "not".to_string(), Expr::Like(Like { negated, case_insensitive, .. }) => { let name = if *case_insensitive { "ilike" } else { "like" }; if *negated { format!("not {name}") } else { name.to_string() } } Expr::SimilarTo(Like { negated, .. }) => { if *negated { "not similar to".to_string() } else { "similar to".to_string() } } _ => { return Err(py_type_err(format!( "Catch all triggered in get_operator_name: {:?}", &self.expr ))); } }) } pub fn column_name(&self, plan: PyLogicalPlan) -> PyResult { self._column_name(&plan.plan()).map_err(py_runtime_err) } // Expression Function Builder functions pub fn order_by(&self, order_by: Vec) -> PyExprFuncBuilder { self.expr .clone() .order_by(to_sort_expressions(order_by)) .into() } pub fn filter(&self, filter: PyExpr) -> PyExprFuncBuilder { self.expr.clone().filter(filter.expr.clone()).into() } pub fn distinct(&self) -> PyExprFuncBuilder { self.expr.clone().distinct().into() } pub fn null_treatment(&self, null_treatment: NullTreatment) -> PyExprFuncBuilder { self.expr .clone() .null_treatment(Some(null_treatment.into())) .into() } pub fn partition_by(&self, partition_by: Vec) -> PyExprFuncBuilder { let partition_by = partition_by.iter().map(|e| e.expr.clone()).collect(); self.expr.clone().partition_by(partition_by).into() } pub fn window_frame(&self, window_frame: PyWindowFrame) -> PyExprFuncBuilder { self.expr.clone().window_frame(window_frame.into()).into() } #[pyo3(signature = (partition_by=None, window_frame=None, order_by=None, null_treatment=None))] pub fn over( &self, partition_by: Option>, window_frame: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { match &self.expr { Expr::AggregateFunction(agg_fn) => { let window_fn = Expr::WindowFunction(Box::new(WindowFunction::new( WindowFunctionDefinition::AggregateUDF(agg_fn.func.clone()), agg_fn.params.args.clone(), ))); add_builder_fns_to_window( window_fn, partition_by, window_frame, order_by, null_treatment, ) } Expr::WindowFunction(_) => add_builder_fns_to_window( self.expr.clone(), partition_by, window_frame, order_by, null_treatment, ), _ => Err(datafusion::error::DataFusionError::Plan(format!( "Using {} with `over` is not allowed. Must use an aggregate or window function.", self.expr.variant_name() )) .into()), } } /// Serialize this `Expr` to protobuf bytes. /// /// When `ctx` is supplied, encoding routes through the session's /// installed `LogicalExtensionCodec` so user FFI codecs see the /// encode path. Without `ctx` a default-inner Python codec is /// used; Python scalar UDFs still inline when in-band encoding /// lands, non-Python UDFs fall through to the default codec. #[pyo3(signature = (ctx=None))] pub fn to_bytes<'py>( &'py self, py: Python<'py>, ctx: Option, ) -> PyDataFusionResult> { let default_codec; let codec: &dyn datafusion_proto::logical_plan::LogicalExtensionCodec = match ctx { Some(ref ctx) => ctx.logical_codec().as_ref(), None => { default_codec = PythonLogicalCodec::default(); &default_codec } }; let proto = to_proto::serialize_expr(&self.expr, codec) .map_err(|e| PyRuntimeError::new_err(format!("Unable to serialize expr: {e}")))?; let bytes = proto.encode_to_vec(); Ok(PyBytes::new(py, &bytes)) } /// Decode an `Expr` from protobuf bytes against the session's /// function registry and logical codec. #[staticmethod] pub fn from_bytes( ctx: PySessionContext, proto_msg: Bound<'_, PyBytes>, ) -> PyDataFusionResult { let bytes: &[u8] = proto_msg.extract().map_err(Into::::into)?; let proto_expr = datafusion_proto::protobuf::LogicalExprNode::decode(bytes).map_err(|e| { PyRuntimeError::new_err(format!( "Unable to decode expression from serialized bytes: {e}" )) })?; let codec = ctx.logical_codec(); let task_ctx = ctx.ctx.task_ctx(); let expr = from_proto::parse_expr(&proto_expr, task_ctx.as_ref(), codec.as_ref()) .map_err(|e| PyRuntimeError::new_err(format!("Unable to decode expr: {e}")))?; Ok(Self { expr }) } } #[pyclass( from_py_object, frozen, name = "ExprFuncBuilder", module = "datafusion.expr", subclass )] #[derive(Debug, Clone)] pub struct PyExprFuncBuilder { pub builder: ExprFuncBuilder, } impl From for PyExprFuncBuilder { fn from(builder: ExprFuncBuilder) -> Self { Self { builder } } } #[pymethods] impl PyExprFuncBuilder { pub fn order_by(&self, order_by: Vec) -> PyExprFuncBuilder { self.builder .clone() .order_by(to_sort_expressions(order_by)) .into() } pub fn filter(&self, filter: PyExpr) -> PyExprFuncBuilder { self.builder.clone().filter(filter.expr.clone()).into() } pub fn distinct(&self) -> PyExprFuncBuilder { self.builder.clone().distinct().into() } pub fn null_treatment(&self, null_treatment: NullTreatment) -> PyExprFuncBuilder { self.builder .clone() .null_treatment(Some(null_treatment.into())) .into() } pub fn partition_by(&self, partition_by: Vec) -> PyExprFuncBuilder { let partition_by = partition_by.iter().map(|e| e.expr.clone()).collect(); self.builder.clone().partition_by(partition_by).into() } pub fn window_frame(&self, window_frame: PyWindowFrame) -> PyExprFuncBuilder { self.builder .clone() .window_frame(window_frame.into()) .into() } pub fn build(&self) -> PyDataFusionResult { Ok(self.builder.clone().build().map(|expr| expr.into())?) } } impl PyExpr { pub fn _column_name(&self, plan: &LogicalPlan) -> PyDataFusionResult { let field = Self::expr_to_field(&self.expr, plan)?; Ok(field.name().to_owned()) } /// Create a [Field] representing an [Expr], given an input [LogicalPlan] to resolve against pub fn expr_to_field(expr: &Expr, input_plan: &LogicalPlan) -> PyDataFusionResult> { let fields = exprlist_to_fields(std::slice::from_ref(expr), input_plan)?; Ok(fields[0].1.clone()) } fn _types(expr: &Expr) -> PyResult { match expr { Expr::BinaryExpr(BinaryExpr { left: _, op, right: _, }) => match op { Operator::Eq | Operator::NotEq | Operator::Lt | Operator::LtEq | Operator::Gt | Operator::GtEq | Operator::And | Operator::Or | Operator::IsDistinctFrom | Operator::IsNotDistinctFrom | Operator::RegexMatch | Operator::RegexIMatch | Operator::RegexNotMatch | Operator::RegexNotIMatch | Operator::LikeMatch | Operator::ILikeMatch | Operator::NotLikeMatch | Operator::NotILikeMatch => DataTypeMap::map_from_arrow_type(&DataType::Boolean), Operator::Plus | Operator::Minus | Operator::Multiply | Operator::Modulo => { DataTypeMap::map_from_arrow_type(&DataType::Int64) } Operator::Divide => DataTypeMap::map_from_arrow_type(&DataType::Float64), Operator::StringConcat => DataTypeMap::map_from_arrow_type(&DataType::Utf8), Operator::BitwiseShiftLeft | Operator::BitwiseShiftRight | Operator::BitwiseXor | Operator::BitwiseAnd | Operator::BitwiseOr => DataTypeMap::map_from_arrow_type(&DataType::Binary), Operator::AtArrow | Operator::ArrowAt | Operator::Arrow | Operator::LongArrow | Operator::HashArrow | Operator::HashLongArrow | Operator::AtAt | Operator::IntegerDivide | Operator::HashMinus | Operator::AtQuestion | Operator::Question | Operator::QuestionAnd | Operator::QuestionPipe | Operator::Colon => Err(py_type_err(format!("Unsupported expr: ${op}"))), }, Expr::Cast(Cast { expr: _, field }) => { DataTypeMap::map_from_arrow_type(field.data_type()) } Expr::Literal(scalar_value, _) => DataTypeMap::map_from_scalar_value(scalar_value), _ => Err(py_type_err(format!( "Non Expr::Literal encountered in types: {expr:?}" ))), } } } /// Initializes the `expr` module to match the pattern of `datafusion-expr` https://docs.rs/datafusion-expr/latest/datafusion_expr/ pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/expr/000077500000000000000000000000001521727353300224615ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/src/expr/aggregate.rs000066400000000000000000000120571521727353300247620ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::common::DataFusionError; use datafusion::logical_expr::Expr; use datafusion::logical_expr::expr::{AggregateFunction, AggregateFunctionParams, Alias}; use datafusion::logical_expr::logical_plan::Aggregate; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::errors::py_type_err; use crate::expr::PyExpr; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Aggregate", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyAggregate { aggregate: Aggregate, } impl From for PyAggregate { fn from(aggregate: Aggregate) -> PyAggregate { PyAggregate { aggregate } } } impl TryFrom for Aggregate { type Error = DataFusionError; fn try_from(agg: PyAggregate) -> Result { Ok(agg.aggregate) } } impl Display for PyAggregate { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Aggregate \nGroupBy(s): {:?} \nAggregates(s): {:?} \nInput: {:?} \nProjected Schema: {:?}", &self.aggregate.group_expr, &self.aggregate.aggr_expr, self.aggregate.input, self.aggregate.schema ) } } #[pymethods] impl PyAggregate { /// Retrieves the grouping expressions for this `Aggregate` fn group_by_exprs(&self) -> PyResult> { Ok(self .aggregate .group_expr .iter() .map(|e| PyExpr::from(e.clone())) .collect()) } /// Retrieves the aggregate expressions for this `Aggregate` fn aggregate_exprs(&self) -> PyResult> { Ok(self .aggregate .aggr_expr .iter() .map(|e| PyExpr::from(e.clone())) .collect()) } /// Returns the inner Aggregate Expr(s) pub fn agg_expressions(&self) -> PyResult> { Ok(self .aggregate .aggr_expr .iter() .map(|e| PyExpr::from(e.clone())) .collect()) } pub fn agg_func_name(&self, expr: PyExpr) -> PyResult { Self::_agg_func_name(&expr.expr) } pub fn aggregation_arguments(&self, expr: PyExpr) -> PyResult> { self._aggregation_arguments(&expr.expr) } // Retrieves the input `LogicalPlan` to this `Aggregate` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } // Resulting Schema for this `Aggregate` node instance fn schema(&self) -> PyDFSchema { (*self.aggregate.schema).clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("Aggregate({self})")) } } impl PyAggregate { #[allow(clippy::only_used_in_recursion)] fn _aggregation_arguments(&self, expr: &Expr) -> PyResult> { match expr { // TODO: This Alias logic seems to be returning some strange results that we should investigate Expr::Alias(Alias { expr, .. }) => self._aggregation_arguments(expr.as_ref()), Expr::AggregateFunction(AggregateFunction { func: _, params: AggregateFunctionParams { args, .. }, .. }) => Ok(args.iter().map(|e| PyExpr::from(e.clone())).collect()), _ => Err(py_type_err( "Encountered a non Aggregate type in aggregation_arguments", )), } } fn _agg_func_name(expr: &Expr) -> PyResult { match expr { Expr::Alias(Alias { expr, .. }) => Self::_agg_func_name(expr.as_ref()), Expr::AggregateFunction(AggregateFunction { func, .. }) => Ok(func.name().to_owned()), _ => Err(py_type_err( "Encountered a non Aggregate type in agg_func_name", )), } } } impl LogicalNode for PyAggregate { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.aggregate.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/aggregate_expr.rs000066400000000000000000000046751521727353300260270ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{Display, Formatter}; use datafusion::logical_expr::expr::AggregateFunction; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "AggregateFunction", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyAggregateFunction { aggr: AggregateFunction, } impl From for AggregateFunction { fn from(aggr: PyAggregateFunction) -> Self { aggr.aggr } } impl From for PyAggregateFunction { fn from(aggr: AggregateFunction) -> PyAggregateFunction { PyAggregateFunction { aggr } } } impl Display for PyAggregateFunction { fn fmt(&self, f: &mut Formatter) -> std::fmt::Result { let args: Vec = self .aggr .params .args .iter() .map(|expr| expr.to_string()) .collect(); write!(f, "{}({})", self.aggr.func.name(), args.join(", ")) } } #[pymethods] impl PyAggregateFunction { /// Get the aggregate type, such as "MIN", or "MAX" fn aggregate_type(&self) -> String { self.aggr.func.name().to_string() } /// is this a distinct aggregate such as `COUNT(DISTINCT expr)` fn is_distinct(&self) -> bool { self.aggr.params.distinct } /// Get the arguments to the aggregate function fn args(&self) -> Vec { self.aggr .params .args .iter() .map(|expr| PyExpr::from(expr.clone())) .collect() } /// Get a String representation of this column fn __repr__(&self) -> String { format!("{self}") } } apache-datafusion-python-54.0.0/crates/core/src/expr/alias.rs000066400000000000000000000036171521727353300241270ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::Alias; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "Alias", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyAlias { alias: Alias, } impl From for PyAlias { fn from(alias: Alias) -> Self { Self { alias } } } impl From for Alias { fn from(py_alias: PyAlias) -> Self { py_alias.alias } } impl Display for PyAlias { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Alias \nExpr: `{:?}` \nAlias Name: `{}`", &self.alias.expr, &self.alias.name ) } } #[pymethods] impl PyAlias { /// Retrieve the "name" of the alias fn alias(&self) -> PyResult { Ok(self.alias.name.clone()) } fn expr(&self) -> PyResult { Ok((*self.alias.expr.clone()).into()) } /// Get a String representation of this column fn __repr__(&self) -> String { format!("{self}") } } apache-datafusion-python-54.0.0/crates/core/src/expr/analyze.rs000066400000000000000000000046051521727353300244770ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::Analyze; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Analyze", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyAnalyze { analyze: Analyze, } impl PyAnalyze { pub fn new(analyze: Analyze) -> Self { Self { analyze } } } impl From for PyAnalyze { fn from(analyze: Analyze) -> PyAnalyze { PyAnalyze { analyze } } } impl From for Analyze { fn from(analyze: PyAnalyze) -> Self { analyze.analyze } } impl Display for PyAnalyze { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "Analyze Table") } } #[pymethods] impl PyAnalyze { fn verbose(&self) -> PyResult { Ok(self.analyze.verbose) } fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Analyze` node instance fn schema(&self) -> PyResult { Ok((*self.analyze.schema).clone().into()) } fn __repr__(&self) -> PyResult { Ok(format!("Analyze({self})")) } } impl LogicalNode for PyAnalyze { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.analyze.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/between.rs000066400000000000000000000041541521727353300244640ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::Between; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "Between", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyBetween { between: Between, } impl From for Between { fn from(between: PyBetween) -> Self { between.between } } impl From for PyBetween { fn from(between: Between) -> PyBetween { PyBetween { between } } } impl Display for PyBetween { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Between Expr: {:?} Negated: {:?} Low: {:?} High: {:?}", &self.between.expr, &self.between.negated, &self.between.low, &self.between.high ) } } #[pymethods] impl PyBetween { fn expr(&self) -> PyResult { Ok((*self.between.expr).clone().into()) } fn negated(&self) -> PyResult { Ok(self.between.negated) } fn low(&self) -> PyResult { Ok((*self.between.low).clone().into()) } fn high(&self) -> PyResult { Ok((*self.between.high).clone().into()) } fn __repr__(&self) -> String { format!("{self}") } } apache-datafusion-python-54.0.0/crates/core/src/expr/binary_expr.rs000066400000000000000000000032221521727353300253500ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::BinaryExpr; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "BinaryExpr", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyBinaryExpr { expr: BinaryExpr, } impl From for BinaryExpr { fn from(expr: PyBinaryExpr) -> Self { expr.expr } } impl From for PyBinaryExpr { fn from(expr: BinaryExpr) -> PyBinaryExpr { PyBinaryExpr { expr } } } #[pymethods] impl PyBinaryExpr { fn left(&self) -> PyExpr { self.expr.left.as_ref().clone().into() } fn right(&self) -> PyExpr { self.expr.right.as_ref().clone().into() } fn op(&self) -> String { format!("{}", self.expr.op) } fn __repr__(&self) -> PyResult { Ok(format!("{}", self.expr)) } } apache-datafusion-python-54.0.0/crates/core/src/expr/bool_expr.rs000066400000000000000000000152031521727353300250210ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::Expr; use pyo3::prelude::*; use super::PyExpr; #[pyclass( from_py_object, frozen, name = "Not", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyNot { expr: Expr, } impl PyNot { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyNot { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Not Expr: {}", &self.expr ) } } #[pymethods] impl PyNot { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsNotNull", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsNotNull { expr: Expr, } impl PyIsNotNull { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsNotNull { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsNotNull Expr: {}", &self.expr ) } } #[pymethods] impl PyIsNotNull { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsNull", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsNull { expr: Expr, } impl PyIsNull { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsNull { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsNull Expr: {}", &self.expr ) } } #[pymethods] impl PyIsNull { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsTrue", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsTrue { expr: Expr, } impl PyIsTrue { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsTrue { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsTrue Expr: {}", &self.expr ) } } #[pymethods] impl PyIsTrue { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsFalse", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsFalse { expr: Expr, } impl PyIsFalse { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsFalse { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsFalse Expr: {}", &self.expr ) } } #[pymethods] impl PyIsFalse { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsUnknown", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsUnknown { expr: Expr, } impl PyIsUnknown { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsUnknown { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsUnknown Expr: {}", &self.expr ) } } #[pymethods] impl PyIsUnknown { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsNotTrue", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsNotTrue { expr: Expr, } impl PyIsNotTrue { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsNotTrue { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsNotTrue Expr: {}", &self.expr ) } } #[pymethods] impl PyIsNotTrue { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsNotFalse", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsNotFalse { expr: Expr, } impl PyIsNotFalse { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsNotFalse { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsNotFalse Expr: {}", &self.expr ) } } #[pymethods] impl PyIsNotFalse { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "IsNotUnknown", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyIsNotUnknown { expr: Expr, } impl PyIsNotUnknown { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyIsNotUnknown { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "IsNotUnknown Expr: {}", &self.expr ) } } #[pymethods] impl PyIsNotUnknown { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } #[pyclass( from_py_object, frozen, name = "Negative", module = "datafusion.expr", subclass )] #[derive(Clone, Debug)] pub struct PyNegative { expr: Expr, } impl PyNegative { pub fn new(expr: Expr) -> Self { Self { expr } } } impl Display for PyNegative { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Negative Expr: {}", &self.expr ) } } #[pymethods] impl PyNegative { fn expr(&self) -> PyResult { Ok(self.expr.clone().into()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/case.rs000066400000000000000000000032661521727353300237510ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::Case; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "Case", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCase { case: Case, } impl From for Case { fn from(case: PyCase) -> Self { case.case } } impl From for PyCase { fn from(case: Case) -> PyCase { PyCase { case } } } #[pymethods] impl PyCase { fn expr(&self) -> Option { self.case.expr.as_ref().map(|e| (**e).clone().into()) } fn when_then_expr(&self) -> Vec<(PyExpr, PyExpr)> { self.case .when_then_expr .iter() .map(|e| ((*e.0).clone().into(), (*e.1).clone().into())) .collect() } fn else_expr(&self) -> Option { self.case.else_expr.as_ref().map(|e| (**e).clone().into()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/cast.rs000066400000000000000000000041551521727353300237660ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::{Cast, TryCast}; use pyo3::prelude::*; use crate::common::data_type::PyDataType; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "Cast", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCast { cast: Cast, } impl From for Cast { fn from(cast: PyCast) -> Self { cast.cast } } impl From for PyCast { fn from(cast: Cast) -> PyCast { PyCast { cast } } } #[pymethods] impl PyCast { fn expr(&self) -> PyResult { Ok((*self.cast.expr).clone().into()) } fn data_type(&self) -> PyResult { Ok(self.cast.field.data_type().clone().into()) } } #[pyclass(from_py_object, name = "TryCast", module = "datafusion.expr", subclass)] #[derive(Clone)] pub struct PyTryCast { try_cast: TryCast, } impl From for TryCast { fn from(try_cast: PyTryCast) -> Self { try_cast.try_cast } } impl From for PyTryCast { fn from(try_cast: TryCast) -> PyTryCast { PyTryCast { try_cast } } } #[pymethods] impl PyTryCast { fn expr(&self) -> PyResult { Ok((*self.try_cast.expr).clone().into()) } fn data_type(&self) -> PyResult { Ok(self.try_cast.field.data_type().clone().into()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/column.rs000066400000000000000000000033051521727353300243250ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::common::Column; use pyo3::prelude::*; #[pyclass( from_py_object, frozen, name = "Column", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyColumn { pub col: Column, } impl PyColumn { pub fn new(col: Column) -> Self { Self { col } } } impl From for PyColumn { fn from(col: Column) -> PyColumn { PyColumn { col } } } #[pymethods] impl PyColumn { /// Get the column name fn name(&self) -> String { self.col.name.clone() } /// Get the column relation fn relation(&self) -> Option { self.col.relation.as_ref().map(|r| format!("{r}")) } /// Get the fully-qualified column name fn qualified_name(&self) -> String { self.col.flat_name() } /// Get a String representation of this column fn __repr__(&self) -> String { self.qualified_name() } } apache-datafusion-python-54.0.0/crates/core/src/expr/conditional_expr.rs000066400000000000000000000045571521727353300264030ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::conditional_expressions::CaseBuilder; use datafusion::prelude::Expr; use pyo3::prelude::*; use crate::errors::PyDataFusionResult; use crate::expr::PyExpr; // TODO(tsaucer) replace this all with CaseBuilder after it implements Clone #[derive(Clone, Debug)] #[pyclass( from_py_object, name = "CaseBuilder", module = "datafusion.expr", subclass, frozen )] pub struct PyCaseBuilder { expr: Option, when: Vec, then: Vec, } #[pymethods] impl PyCaseBuilder { #[new] pub fn new(expr: Option) -> Self { Self { expr: expr.map(Into::into), when: vec![], then: vec![], } } pub fn when(&self, when: PyExpr, then: PyExpr) -> PyCaseBuilder { let mut case_builder = self.clone(); case_builder.when.push(when.into()); case_builder.then.push(then.into()); case_builder } fn otherwise(&self, else_expr: PyExpr) -> PyDataFusionResult { let case_builder = CaseBuilder::new( self.expr.clone().map(Box::new), self.when.clone(), self.then.clone(), Some(Box::new(else_expr.into())), ); let expr = case_builder.end()?; Ok(expr.into()) } fn end(&self) -> PyDataFusionResult { let case_builder = CaseBuilder::new( self.expr.clone().map(Box::new), self.when.clone(), self.then.clone(), None, ); let expr = case_builder.end()?; Ok(expr.into()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/copy_to.rs000066400000000000000000000070421521727353300245060ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::common::file_options::file_type::FileType; use datafusion::logical_expr::dml::CopyTo; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CopyTo", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCopyTo { copy: CopyTo, } impl From for CopyTo { fn from(copy: PyCopyTo) -> Self { copy.copy } } impl From for PyCopyTo { fn from(copy: CopyTo) -> PyCopyTo { PyCopyTo { copy } } } impl Display for PyCopyTo { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "CopyTo: {:?}", self.copy.output_url) } } impl LogicalNode for PyCopyTo { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.copy.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyCopyTo { #[new] pub fn new( input: PyLogicalPlan, output_url: String, partition_by: Vec, file_type: PyFileType, options: HashMap, ) -> Self { PyCopyTo { copy: CopyTo::new( input.plan(), output_url, partition_by, file_type.file_type, options, ), } } fn input(&self) -> PyLogicalPlan { PyLogicalPlan::from((*self.copy.input).clone()) } fn output_url(&self) -> String { self.copy.output_url.clone() } fn partition_by(&self) -> Vec { self.copy.partition_by.clone() } fn file_type(&self) -> PyFileType { PyFileType { file_type: self.copy.file_type.clone(), } } fn options(&self) -> HashMap { self.copy.options.clone() } fn __repr__(&self) -> PyResult { Ok(format!("CopyTo({self})")) } fn __name__(&self) -> PyResult { Ok("CopyTo".to_string()) } } #[pyclass( from_py_object, frozen, name = "FileType", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyFileType { file_type: Arc, } impl Display for PyFileType { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "FileType: {}", self.file_type) } } #[pymethods] impl PyFileType { fn __repr__(&self) -> PyResult { Ok(format!("FileType({self})")) } fn __name__(&self) -> PyResult { Ok("FileType".to_string()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_catalog.rs000066400000000000000000000053631521727353300257730ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::CreateCatalog; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateCatalog", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateCatalog { create: CreateCatalog, } impl From for CreateCatalog { fn from(create: PyCreateCatalog) -> Self { create.create } } impl From for PyCreateCatalog { fn from(create: CreateCatalog) -> PyCreateCatalog { PyCreateCatalog { create } } } impl Display for PyCreateCatalog { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "CreateCatalog: {:?}", self.create.catalog_name) } } #[pymethods] impl PyCreateCatalog { #[new] pub fn new( catalog_name: String, if_not_exists: bool, schema: PyDFSchema, ) -> PyResult { Ok(PyCreateCatalog { create: CreateCatalog { catalog_name, if_not_exists, schema: Arc::new(schema.into()), }, }) } pub fn catalog_name(&self) -> String { self.create.catalog_name.clone() } pub fn if_not_exists(&self) -> bool { self.create.if_not_exists } pub fn schema(&self) -> PyDFSchema { (*self.create.schema).clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("CreateCatalog({self})")) } fn __name__(&self) -> PyResult { Ok("CreateCatalog".to_string()) } } impl LogicalNode for PyCreateCatalog { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_catalog_schema.rs000066400000000000000000000055541521727353300273150ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::CreateCatalogSchema; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateCatalogSchema", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateCatalogSchema { create: CreateCatalogSchema, } impl From for CreateCatalogSchema { fn from(create: PyCreateCatalogSchema) -> Self { create.create } } impl From for PyCreateCatalogSchema { fn from(create: CreateCatalogSchema) -> PyCreateCatalogSchema { PyCreateCatalogSchema { create } } } impl Display for PyCreateCatalogSchema { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "CreateCatalogSchema: {:?}", self.create.schema_name) } } #[pymethods] impl PyCreateCatalogSchema { #[new] pub fn new( schema_name: String, if_not_exists: bool, schema: PyDFSchema, ) -> PyResult { Ok(PyCreateCatalogSchema { create: CreateCatalogSchema { schema_name, if_not_exists, schema: Arc::new(schema.into()), }, }) } pub fn schema_name(&self) -> String { self.create.schema_name.clone() } pub fn if_not_exists(&self) -> bool { self.create.if_not_exists } pub fn schema(&self) -> PyDFSchema { (*self.create.schema).clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("CreateCatalogSchema({self})")) } fn __name__(&self) -> PyResult { Ok("CreateCatalogSchema".to_string()) } } impl LogicalNode for PyCreateCatalogSchema { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_external_table.rs000066400000000000000000000125601521727353300273470ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::CreateExternalTable; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use super::sort_expr::PySortExpr; use crate::common::df_schema::PyDFSchema; use crate::common::schema::PyConstraints; use crate::expr::PyExpr; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateExternalTable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateExternalTable { create: CreateExternalTable, } impl From for CreateExternalTable { fn from(create: PyCreateExternalTable) -> Self { create.create } } impl From for PyCreateExternalTable { fn from(create: CreateExternalTable) -> PyCreateExternalTable { PyCreateExternalTable { create } } } impl Display for PyCreateExternalTable { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "CreateExternalTable: {:?}{}", self.create.name, self.create.constraints ) } } #[pymethods] impl PyCreateExternalTable { #[allow(clippy::too_many_arguments)] #[new] #[pyo3(signature = (schema, name, location, file_type, table_partition_cols, if_not_exists, or_replace, temporary, order_exprs, unbounded, options, constraints, column_defaults, definition=None))] pub fn new( schema: PyDFSchema, name: String, location: String, file_type: String, table_partition_cols: Vec, if_not_exists: bool, or_replace: bool, temporary: bool, order_exprs: Vec>, unbounded: bool, options: HashMap, constraints: PyConstraints, column_defaults: HashMap, definition: Option, ) -> Self { let create = CreateExternalTable { schema: Arc::new(schema.into()), name: name.into(), location, file_type, table_partition_cols, if_not_exists, or_replace, temporary, definition, order_exprs: order_exprs .into_iter() .map(|vec| vec.into_iter().map(|s| s.into()).collect::>()) .collect::>(), unbounded, options, constraints: constraints.constraints, column_defaults: column_defaults .into_iter() .map(|(k, v)| (k, v.into())) .collect(), }; PyCreateExternalTable { create } } pub fn schema(&self) -> PyDFSchema { (*self.create.schema).clone().into() } pub fn name(&self) -> PyResult { Ok(self.create.name.to_string()) } pub fn location(&self) -> String { self.create.location.clone() } pub fn file_type(&self) -> String { self.create.file_type.clone() } pub fn table_partition_cols(&self) -> Vec { self.create.table_partition_cols.clone() } pub fn if_not_exists(&self) -> bool { self.create.if_not_exists } pub fn temporary(&self) -> bool { self.create.temporary } pub fn definition(&self) -> Option { self.create.definition.clone() } pub fn order_exprs(&self) -> Vec> { self.create .order_exprs .iter() .map(|vec| vec.iter().map(|s| s.clone().into()).collect()) .collect() } pub fn unbounded(&self) -> bool { self.create.unbounded } pub fn options(&self) -> HashMap { self.create.options.clone() } pub fn constraints(&self) -> PyConstraints { PyConstraints { constraints: self.create.constraints.clone(), } } pub fn column_defaults(&self) -> HashMap { self.create .column_defaults .iter() .map(|(k, v)| (k.clone(), v.clone().into())) .collect() } fn __repr__(&self) -> PyResult { Ok(format!("CreateExternalTable({self})")) } fn __name__(&self) -> PyResult { Ok("CreateExternalTable".to_string()) } } impl LogicalNode for PyCreateExternalTable { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_function.rs000066400000000000000000000124241521727353300262020ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::{ CreateFunction, CreateFunctionBody, OperateFunctionArg, Volatility, }; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::PyExpr; use super::logical_node::LogicalNode; use crate::common::data_type::PyDataType; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateFunction", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateFunction { create: CreateFunction, } impl From for CreateFunction { fn from(create: PyCreateFunction) -> Self { create.create } } impl From for PyCreateFunction { fn from(create: CreateFunction) -> PyCreateFunction { PyCreateFunction { create } } } impl Display for PyCreateFunction { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "CreateFunction: name {:?}", self.create.name) } } #[pyclass( from_py_object, frozen, name = "OperateFunctionArg", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyOperateFunctionArg { arg: OperateFunctionArg, } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "Volatility", module = "datafusion.expr" )] pub enum PyVolatility { Immutable, Stable, Volatile, } #[pyclass( from_py_object, frozen, name = "CreateFunctionBody", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateFunctionBody { body: CreateFunctionBody, } #[pymethods] impl PyCreateFunctionBody { pub fn language(&self) -> Option { self.body .language .as_ref() .map(|language| language.to_string()) } pub fn behavior(&self) -> Option { self.body.behavior.as_ref().map(|behavior| match behavior { Volatility::Immutable => PyVolatility::Immutable, Volatility::Stable => PyVolatility::Stable, Volatility::Volatile => PyVolatility::Volatile, }) } pub fn function_body(&self) -> Option { self.body .function_body .as_ref() .map(|function_body| function_body.clone().into()) } } #[pymethods] impl PyCreateFunction { #[new] #[pyo3(signature = (or_replace, temporary, name, params, schema, return_type=None, args=None))] pub fn new( or_replace: bool, temporary: bool, name: String, params: PyCreateFunctionBody, schema: PyDFSchema, return_type: Option, args: Option>, ) -> Self { PyCreateFunction { create: CreateFunction { or_replace, temporary, name, args: args.map(|args| args.into_iter().map(|arg| arg.arg).collect()), return_type: return_type.map(|return_type| return_type.data_type), params: params.body, schema: Arc::new(schema.into()), }, } } pub fn or_replace(&self) -> bool { self.create.or_replace } pub fn temporary(&self) -> bool { self.create.temporary } pub fn name(&self) -> String { self.create.name.clone() } pub fn params(&self) -> PyCreateFunctionBody { PyCreateFunctionBody { body: self.create.params.clone(), } } pub fn schema(&self) -> PyDFSchema { (*self.create.schema).clone().into() } pub fn return_type(&self) -> Option { self.create .return_type .as_ref() .map(|return_type| return_type.clone().into()) } pub fn args(&self) -> Option> { self.create.args.as_ref().map(|args| { args.iter() .map(|arg| PyOperateFunctionArg { arg: arg.clone() }) .collect() }) } fn __repr__(&self) -> PyResult { Ok(format!("CreateFunction({self})")) } fn __name__(&self) -> PyResult { Ok("CreateFunction".to_string()) } } impl LogicalNode for PyCreateFunction { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_index.rs000066400000000000000000000067431521727353300254730ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::CreateIndex; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use super::sort_expr::PySortExpr; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateIndex", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateIndex { create: CreateIndex, } impl From for CreateIndex { fn from(create: PyCreateIndex) -> Self { create.create } } impl From for PyCreateIndex { fn from(create: CreateIndex) -> PyCreateIndex { PyCreateIndex { create } } } impl Display for PyCreateIndex { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "CreateIndex: {:?}", self.create.name) } } #[pymethods] impl PyCreateIndex { #[new] #[pyo3(signature = (table, columns, unique, if_not_exists, schema, name=None, using=None))] pub fn new( table: String, columns: Vec, unique: bool, if_not_exists: bool, schema: PyDFSchema, name: Option, using: Option, ) -> PyResult { Ok(PyCreateIndex { create: CreateIndex { name, table: table.into(), using, columns: columns.iter().map(|c| c.clone().into()).collect(), unique, if_not_exists, schema: Arc::new(schema.into()), }, }) } pub fn name(&self) -> Option { self.create.name.clone() } pub fn table(&self) -> PyResult { Ok(self.create.table.to_string()) } pub fn using(&self) -> Option { self.create.using.clone() } pub fn columns(&self) -> Vec { self.create .columns .iter() .map(|c| c.clone().into()) .collect() } pub fn unique(&self) -> bool { self.create.unique } pub fn if_not_exists(&self) -> bool { self.create.if_not_exists } pub fn schema(&self) -> PyDFSchema { (*self.create.schema).clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("CreateIndex({self})")) } fn __name__(&self) -> PyResult { Ok("CreateIndex".to_string()) } } impl LogicalNode for PyCreateIndex { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_memory_table.rs000066400000000000000000000054041521727353300270340ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::CreateMemoryTable; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateMemoryTable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateMemoryTable { create: CreateMemoryTable, } impl From for CreateMemoryTable { fn from(create: PyCreateMemoryTable) -> Self { create.create } } impl From for PyCreateMemoryTable { fn from(create: CreateMemoryTable) -> PyCreateMemoryTable { PyCreateMemoryTable { create } } } impl Display for PyCreateMemoryTable { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "CreateMemoryTable Name: {:?} Input: {:?} if_not_exists: {:?} or_replace: {:?}", &self.create.name, &self.create.input, &self.create.if_not_exists, &self.create.or_replace, ) } } #[pymethods] impl PyCreateMemoryTable { fn name(&self) -> PyResult { Ok(self.create.name.to_string()) } fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } fn if_not_exists(&self) -> bool { self.create.if_not_exists } fn or_replace(&self) -> bool { self.create.or_replace } fn __repr__(&self) -> PyResult { Ok(format!("CreateMemoryTable({self})")) } fn __name__(&self) -> PyResult { Ok("CreateMemoryTable".to_string()) } } impl LogicalNode for PyCreateMemoryTable { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.create.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/create_view.rs000066400000000000000000000060101521727353300253210ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::{CreateView, DdlStatement, LogicalPlan}; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::errors::py_type_err; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "CreateView", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyCreateView { create: CreateView, } impl From for CreateView { fn from(create: PyCreateView) -> Self { create.create } } impl From for PyCreateView { fn from(create: CreateView) -> PyCreateView { PyCreateView { create } } } impl Display for PyCreateView { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "CreateView name: {:?} input: {:?} or_replace: {:?} definition: {:?}", &self.create.name, &self.create.input, &self.create.or_replace, &self.create.definition, ) } } #[pymethods] impl PyCreateView { fn name(&self) -> PyResult { Ok(self.create.name.to_string()) } fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } fn or_replace(&self) -> bool { self.create.or_replace } fn definition(&self) -> PyResult> { Ok(self.create.definition.clone()) } fn __repr__(&self) -> PyResult { Ok(format!("CreateView({self})")) } fn __name__(&self) -> PyResult { Ok("CreateView".to_string()) } } impl LogicalNode for PyCreateView { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.create.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } impl TryFrom for PyCreateView { type Error = PyErr; fn try_from(logical_plan: LogicalPlan) -> Result { match logical_plan { LogicalPlan::Ddl(DdlStatement::CreateView(create)) => Ok(PyCreateView { create }), _ => Err(py_type_err("unexpected plan")), } } } apache-datafusion-python-54.0.0/crates/core/src/expr/describe_table.rs000066400000000000000000000052311521727353300257570ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use arrow::datatypes::Schema; use arrow::pyarrow::PyArrowType; use datafusion::logical_expr::DescribeTable; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "DescribeTable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDescribeTable { describe: DescribeTable, } impl Display for PyDescribeTable { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "DescribeTable") } } #[pymethods] impl PyDescribeTable { #[new] fn new(schema: PyArrowType, output_schema: PyDFSchema) -> Self { Self { describe: DescribeTable { schema: Arc::new(schema.0), output_schema: Arc::new(output_schema.into()), }, } } pub fn schema(&self) -> PyArrowType { (*self.describe.schema).clone().into() } pub fn output_schema(&self) -> PyDFSchema { (*self.describe.output_schema).clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("DescribeTable({self})")) } fn __name__(&self) -> PyResult { Ok("DescribeTable".to_string()) } } impl From for DescribeTable { fn from(describe: PyDescribeTable) -> Self { describe.describe } } impl From for PyDescribeTable { fn from(describe: DescribeTable) -> PyDescribeTable { PyDescribeTable { describe } } } impl LogicalNode for PyDescribeTable { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/distinct.rs000066400000000000000000000053431521727353300246550ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::Distinct; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Distinct", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDistinct { distinct: Distinct, } impl From for Distinct { fn from(distinct: PyDistinct) -> Self { distinct.distinct } } impl From for PyDistinct { fn from(distinct: Distinct) -> PyDistinct { PyDistinct { distinct } } } impl Display for PyDistinct { fn fmt(&self, f: &mut Formatter) -> fmt::Result { match &self.distinct { Distinct::All(input) => write!( f, "Distinct ALL \nInput: {input:?}", ), Distinct::On(distinct_on) => { write!( f, "Distinct ON \nInput: {:?}", distinct_on.input, ) } } } } #[pymethods] impl PyDistinct { /// Retrieves the input `LogicalPlan` to this `Projection` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } fn __repr__(&self) -> PyResult { Ok(format!("Distinct({self})")) } fn __name__(&self) -> PyResult { Ok("Distinct".to_string()) } } impl LogicalNode for PyDistinct { fn inputs(&self) -> Vec { match &self.distinct { Distinct::All(input) => vec![PyLogicalPlan::from(input.as_ref().clone())], Distinct::On(distinct_on) => { vec![PyLogicalPlan::from(distinct_on.input.as_ref().clone())] } } } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/dml.rs000066400000000000000000000100141521727353300235770ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::dml::InsertOp; use datafusion::logical_expr::{DmlStatement, WriteOp}; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::common::schema::PyTableSource; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "DmlStatement", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDmlStatement { dml: DmlStatement, } impl From for DmlStatement { fn from(dml: PyDmlStatement) -> Self { dml.dml } } impl From for PyDmlStatement { fn from(dml: DmlStatement) -> PyDmlStatement { PyDmlStatement { dml } } } impl LogicalNode for PyDmlStatement { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.dml.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyDmlStatement { pub fn table_name(&self) -> PyResult { Ok(self.dml.table_name.to_string()) } pub fn target(&self) -> PyResult { Ok(PyTableSource { table_source: self.dml.target.clone(), }) } pub fn op(&self) -> PyWriteOp { self.dml.op.clone().into() } pub fn input(&self) -> PyLogicalPlan { PyLogicalPlan { plan: self.dml.input.clone(), } } pub fn output_schema(&self) -> PyDFSchema { (*self.dml.output_schema).clone().into() } fn __repr__(&self) -> PyResult { Ok("DmlStatement".to_string()) } fn __name__(&self) -> PyResult { Ok("DmlStatement".to_string()) } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, eq, eq_int, name = "WriteOp", module = "datafusion.expr" )] pub enum PyWriteOp { Append, Overwrite, Replace, Update, Delete, Ctas, Truncate, } impl From for PyWriteOp { fn from(write_op: WriteOp) -> Self { match write_op { WriteOp::Insert(InsertOp::Append) => PyWriteOp::Append, WriteOp::Insert(InsertOp::Overwrite) => PyWriteOp::Overwrite, WriteOp::Insert(InsertOp::Replace) => PyWriteOp::Replace, WriteOp::Update => PyWriteOp::Update, WriteOp::Delete => PyWriteOp::Delete, WriteOp::Ctas => PyWriteOp::Ctas, WriteOp::Truncate => PyWriteOp::Truncate, } } } impl From for WriteOp { fn from(py: PyWriteOp) -> Self { match py { PyWriteOp::Append => WriteOp::Insert(InsertOp::Append), PyWriteOp::Overwrite => WriteOp::Insert(InsertOp::Overwrite), PyWriteOp::Replace => WriteOp::Insert(InsertOp::Replace), PyWriteOp::Update => WriteOp::Update, PyWriteOp::Delete => WriteOp::Delete, PyWriteOp::Ctas => WriteOp::Ctas, PyWriteOp::Truncate => WriteOp::Truncate, } } } #[pymethods] impl PyWriteOp { fn name(&self) -> String { let write_op: WriteOp = self.clone().into(); write_op.name().to_string() } } apache-datafusion-python-54.0.0/crates/core/src/expr/drop_catalog_schema.rs000066400000000000000000000066721521727353300270200ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::common::SchemaReference; use datafusion::logical_expr::DropCatalogSchema; use datafusion::sql::TableReference; use pyo3::IntoPyObjectExt; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "DropCatalogSchema", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDropCatalogSchema { drop: DropCatalogSchema, } impl From for DropCatalogSchema { fn from(drop: PyDropCatalogSchema) -> Self { drop.drop } } impl From for PyDropCatalogSchema { fn from(drop: DropCatalogSchema) -> PyDropCatalogSchema { PyDropCatalogSchema { drop } } } impl Display for PyDropCatalogSchema { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "DropCatalogSchema") } } fn parse_schema_reference(name: String) -> PyResult { match name.into() { TableReference::Bare { table } => Ok(SchemaReference::Bare { schema: table }), TableReference::Partial { schema, table } => Ok(SchemaReference::Full { schema: table, catalog: schema, }), TableReference::Full { catalog: _, schema: _, table: _, } => Err(PyErr::new::( "Invalid schema specifier (has 3 parts)".to_string(), )), } } #[pymethods] impl PyDropCatalogSchema { #[new] fn new(name: String, schema: PyDFSchema, if_exists: bool, cascade: bool) -> PyResult { let name = parse_schema_reference(name)?; Ok(PyDropCatalogSchema { drop: DropCatalogSchema { name, schema: Arc::new(schema.into()), if_exists, cascade, }, }) } fn name(&self) -> PyResult { Ok(self.drop.name.to_string()) } fn schema(&self) -> PyDFSchema { (*self.drop.schema).clone().into() } fn if_exists(&self) -> PyResult { Ok(self.drop.if_exists) } fn cascade(&self) -> PyResult { Ok(self.drop.cascade) } fn __repr__(&self) -> PyResult { Ok(format!("DropCatalogSchema({self})")) } } impl LogicalNode for PyDropCatalogSchema { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/drop_function.rs000066400000000000000000000051341521727353300257030ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::DropFunction; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "DropFunction", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDropFunction { drop: DropFunction, } impl From for DropFunction { fn from(drop: PyDropFunction) -> Self { drop.drop } } impl From for PyDropFunction { fn from(drop: DropFunction) -> PyDropFunction { PyDropFunction { drop } } } impl Display for PyDropFunction { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "DropFunction") } } #[pymethods] impl PyDropFunction { #[new] fn new(name: String, schema: PyDFSchema, if_exists: bool) -> PyResult { Ok(PyDropFunction { drop: DropFunction { name, schema: Arc::new(schema.into()), if_exists, }, }) } fn name(&self) -> PyResult { Ok(self.drop.name.clone()) } fn schema(&self) -> PyDFSchema { (*self.drop.schema).clone().into() } fn if_exists(&self) -> PyResult { Ok(self.drop.if_exists) } fn __repr__(&self) -> PyResult { Ok(format!("DropFunction({self})")) } fn __name__(&self) -> PyResult { Ok("DropFunction".to_string()) } } impl LogicalNode for PyDropFunction { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/drop_table.rs000066400000000000000000000046121521727353300251450ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::DropTable; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "DropTable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDropTable { drop: DropTable, } impl From for DropTable { fn from(drop: PyDropTable) -> Self { drop.drop } } impl From for PyDropTable { fn from(drop: DropTable) -> PyDropTable { PyDropTable { drop } } } impl Display for PyDropTable { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "DropTable name: {:?} if_exists: {:?} schema: {:?}", &self.drop.name, &self.drop.if_exists, &self.drop.schema, ) } } #[pymethods] impl PyDropTable { fn name(&self) -> PyResult { Ok(self.drop.name.to_string()) } fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } fn if_exists(&self) -> bool { self.drop.if_exists } fn __repr__(&self) -> PyResult { Ok(format!("DropTable({self})")) } fn __name__(&self) -> PyResult { Ok("DropTable".to_string()) } } impl LogicalNode for PyDropTable { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/drop_view.rs000066400000000000000000000052641521727353300250340ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use std::sync::Arc; use datafusion::logical_expr::DropView; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "DropView", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDropView { drop: DropView, } impl From for DropView { fn from(drop: PyDropView) -> Self { drop.drop } } impl From for PyDropView { fn from(drop: DropView) -> PyDropView { PyDropView { drop } } } impl Display for PyDropView { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "DropView: {name:?} if not exist:={if_exists}", name = self.drop.name, if_exists = self.drop.if_exists ) } } #[pymethods] impl PyDropView { #[new] fn new(name: String, schema: PyDFSchema, if_exists: bool) -> PyResult { Ok(PyDropView { drop: DropView { name: name.into(), schema: Arc::new(schema.into()), if_exists, }, }) } fn name(&self) -> PyResult { Ok(self.drop.name.to_string()) } fn schema(&self) -> PyDFSchema { (*self.drop.schema).clone().into() } fn if_exists(&self) -> PyResult { Ok(self.drop.if_exists) } fn __repr__(&self) -> PyResult { Ok(format!("DropView({self})")) } fn __name__(&self) -> PyResult { Ok("DropView".to_string()) } } impl LogicalNode for PyDropView { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/empty_relation.rs000066400000000000000000000051151521727353300260640ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::EmptyRelation; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "EmptyRelation", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyEmptyRelation { empty: EmptyRelation, } impl From for EmptyRelation { fn from(empty_relation: PyEmptyRelation) -> Self { empty_relation.empty } } impl From for PyEmptyRelation { fn from(empty: EmptyRelation) -> PyEmptyRelation { PyEmptyRelation { empty } } } impl Display for PyEmptyRelation { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Empty Relation Produce One Row: {:?} Schema: {:?}", &self.empty.produce_one_row, &self.empty.schema ) } } #[pymethods] impl PyEmptyRelation { fn produce_one_row(&self) -> PyResult { Ok(self.empty.produce_one_row) } /// Resulting Schema for this `EmptyRelation` node instance fn schema(&self) -> PyResult { Ok((*self.empty.schema).clone().into()) } /// Get a String representation of this column fn __repr__(&self) -> String { format!("{self}") } fn __name__(&self) -> PyResult { Ok("EmptyRelation".to_string()) } } impl LogicalNode for PyEmptyRelation { fn inputs(&self) -> Vec { // table scans are leaf nodes and do not have inputs vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/exists.rs000066400000000000000000000025341521727353300243520ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::expr::Exists; use pyo3::prelude::*; use super::subquery::PySubquery; #[pyclass( from_py_object, frozen, name = "Exists", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyExists { exists: Exists, } impl From for PyExists { fn from(exists: Exists) -> Self { PyExists { exists } } } #[pymethods] impl PyExists { fn subquery(&self) -> PySubquery { self.exists.subquery.clone().into() } fn negated(&self) -> bool { self.exists.negated } } apache-datafusion-python-54.0.0/crates/core/src/expr/explain.rs000066400000000000000000000063721521727353300244770ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::LogicalPlan; use datafusion::logical_expr::logical_plan::Explain; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::errors::py_type_err; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Explain", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyExplain { explain: Explain, } impl From for Explain { fn from(explain: PyExplain) -> Self { explain.explain } } impl From for PyExplain { fn from(explain: Explain) -> PyExplain { PyExplain { explain } } } impl Display for PyExplain { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Explain verbose: {:?} plan: {:?} stringified_plans: {:?} schema: {:?} logical_optimization_succeeded: {:?}", &self.explain.verbose, &self.explain.plan, &self.explain.stringified_plans, &self.explain.schema, &self.explain.logical_optimization_succeeded ) } } #[pymethods] impl PyExplain { fn explain_string(&self) -> PyResult> { let mut string_plans: Vec = Vec::new(); for stringified_plan in &self.explain.stringified_plans { string_plans.push((*stringified_plan.plan).clone()); } Ok(string_plans) } fn verbose(&self) -> bool { self.explain.verbose } fn plan(&self) -> PyResult { Ok(PyLogicalPlan::from((*self.explain.plan).clone())) } fn schema(&self) -> PyDFSchema { (*self.explain.schema).clone().into() } fn logical_optimization_succceeded(&self) -> bool { self.explain.logical_optimization_succeeded } } impl TryFrom for PyExplain { type Error = PyErr; fn try_from(logical_plan: LogicalPlan) -> Result { match logical_plan { LogicalPlan::Explain(explain) => Ok(PyExplain { explain }), _ => Err(py_type_err("unexpected plan")), } } } impl LogicalNode for PyExplain { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/extension.rs000066400000000000000000000031511521727353300250430ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::Extension; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Extension", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyExtension { pub node: Extension, } impl From for PyExtension { fn from(node: Extension) -> PyExtension { PyExtension { node } } } #[pymethods] impl PyExtension { fn name(&self) -> PyResult { Ok(self.node.node.name().to_string()) } } impl LogicalNode for PyExtension { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/filter.rs000066400000000000000000000050601521727353300243150ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::Filter; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::PyExpr; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Filter", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyFilter { filter: Filter, } impl From for PyFilter { fn from(filter: Filter) -> PyFilter { PyFilter { filter } } } impl From for Filter { fn from(filter: PyFilter) -> Self { filter.filter } } impl Display for PyFilter { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Filter Predicate: {:?} Input: {:?}", &self.filter.predicate, &self.filter.input ) } } #[pymethods] impl PyFilter { /// Retrieves the predicate expression for this `Filter` fn predicate(&self) -> PyExpr { PyExpr::from(self.filter.predicate.clone()) } /// Retrieves the input `LogicalPlan` to this `Filter` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Filter` node instance fn schema(&self) -> PyDFSchema { self.filter.input.schema().as_ref().clone().into() } fn __repr__(&self) -> String { format!("Filter({self})") } } impl LogicalNode for PyFilter { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.filter.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/grouping_set.rs000066400000000000000000000043131521727353300255350ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::{Expr, GroupingSet}; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "GroupingSet", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyGroupingSet { grouping_set: GroupingSet, } #[pymethods] impl PyGroupingSet { #[staticmethod] #[pyo3(signature = (*exprs))] fn rollup(exprs: Vec) -> PyExpr { Expr::GroupingSet(GroupingSet::Rollup( exprs.into_iter().map(|e| e.expr).collect(), )) .into() } #[staticmethod] #[pyo3(signature = (*exprs))] fn cube(exprs: Vec) -> PyExpr { Expr::GroupingSet(GroupingSet::Cube( exprs.into_iter().map(|e| e.expr).collect(), )) .into() } #[staticmethod] #[pyo3(signature = (*expr_lists))] fn grouping_sets(expr_lists: Vec>) -> PyExpr { Expr::GroupingSet(GroupingSet::GroupingSets( expr_lists .into_iter() .map(|list| list.into_iter().map(|e| e.expr).collect()) .collect(), )) .into() } } impl From for GroupingSet { fn from(grouping_set: PyGroupingSet) -> Self { grouping_set.grouping_set } } impl From for PyGroupingSet { fn from(grouping_set: GroupingSet) -> PyGroupingSet { PyGroupingSet { grouping_set } } } apache-datafusion-python-54.0.0/crates/core/src/expr/higher_order_function.rs000066400000000000000000000046751521727353300274110ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::HigherOrderFunction; use pyo3::prelude::*; use super::PyExpr; #[pyclass( from_py_object, frozen, name = "HigherOrderFunction", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyHigherOrderFunction { higher_order: HigherOrderFunction, } impl From for PyHigherOrderFunction { fn from(higher_order: HigherOrderFunction) -> PyHigherOrderFunction { PyHigherOrderFunction { higher_order } } } impl From for HigherOrderFunction { fn from(higher_order: PyHigherOrderFunction) -> Self { higher_order.higher_order } } impl Display for PyHigherOrderFunction { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "HigherOrderFunction(name={}, args={:?})", self.higher_order.name(), &self.higher_order.args, ) } } #[pymethods] impl PyHigherOrderFunction { /// Name of the higher-order function being invoked. fn name(&self) -> String { self.higher_order.name().to_string() } /// Arguments passed to the higher-order function. Some entries may be /// `Lambda` expressions; others are ordinary value expressions. fn args(&self) -> Vec { self.higher_order .args .iter() .map(|e| PyExpr::from(e.clone())) .collect() } fn __repr__(&self) -> PyResult { Ok(format!("HigherOrderFunction({self})")) } fn __name__(&self) -> PyResult { Ok("HigherOrderFunction".to_string()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/in_list.rs000066400000000000000000000027001521727353300244670ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::expr::InList; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "InList", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyInList { in_list: InList, } impl From for PyInList { fn from(in_list: InList) -> Self { PyInList { in_list } } } #[pymethods] impl PyInList { fn expr(&self) -> PyExpr { (*self.in_list.expr).clone().into() } fn list(&self) -> Vec { self.in_list.list.iter().map(|e| e.clone().into()).collect() } fn negated(&self) -> bool { self.in_list.negated } } apache-datafusion-python-54.0.0/crates/core/src/expr/in_subquery.rs000066400000000000000000000030021521727353300253670ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::expr::InSubquery; use pyo3::prelude::*; use super::PyExpr; use super::subquery::PySubquery; #[pyclass( from_py_object, frozen, name = "InSubquery", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyInSubquery { in_subquery: InSubquery, } impl From for PyInSubquery { fn from(in_subquery: InSubquery) -> Self { PyInSubquery { in_subquery } } } #[pymethods] impl PyInSubquery { fn expr(&self) -> PyExpr { (*self.in_subquery.expr).clone().into() } fn subquery(&self) -> PySubquery { self.in_subquery.subquery.clone().into() } fn negated(&self) -> bool { self.in_subquery.negated } } apache-datafusion-python-54.0.0/crates/core/src/expr/indexed_field.rs000066400000000000000000000043531521727353300256170ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{Display, Formatter}; use datafusion::logical_expr::expr::{GetFieldAccess, GetIndexedField}; use pyo3::prelude::*; use super::literal::PyLiteral; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "GetIndexedField", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyGetIndexedField { indexed_field: GetIndexedField, } impl From for GetIndexedField { fn from(indexed_field: PyGetIndexedField) -> Self { indexed_field.indexed_field } } impl From for PyGetIndexedField { fn from(indexed_field: GetIndexedField) -> PyGetIndexedField { PyGetIndexedField { indexed_field } } } impl Display for PyGetIndexedField { fn fmt(&self, f: &mut Formatter) -> std::fmt::Result { write!( f, "GetIndexedField Expr: {:?} Key: {:?}", &self.indexed_field.expr, &self.indexed_field.field ) } } #[pymethods] impl PyGetIndexedField { fn expr(&self) -> PyResult { Ok((*self.indexed_field.expr).clone().into()) } fn key(&self) -> PyResult { match &self.indexed_field.field { GetFieldAccess::NamedStructField { name, .. } => Ok(name.clone().into()), _ => todo!(), } } /// Get a String representation of this column fn __repr__(&self) -> String { format!("{}", self) } } apache-datafusion-python-54.0.0/crates/core/src/expr/join.rs000066400000000000000000000133261521727353300237730ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::common::NullEquality; use datafusion::logical_expr::logical_plan::{Join, JoinConstraint, JoinType}; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::PyExpr; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[derive(Debug, Clone, PartialEq, Eq, Hash)] #[pyclass(from_py_object, frozen, name = "JoinType", module = "datafusion.expr")] pub struct PyJoinType { join_type: JoinType, } impl From for PyJoinType { fn from(join_type: JoinType) -> PyJoinType { PyJoinType { join_type } } } impl From for JoinType { fn from(join_type: PyJoinType) -> Self { join_type.join_type } } #[pymethods] impl PyJoinType { pub fn is_outer(&self) -> bool { self.join_type.is_outer() } fn __repr__(&self) -> PyResult { Ok(format!("{}", self.join_type)) } } impl Display for PyJoinType { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "{}", self.join_type) } } #[derive(Debug, Clone, Copy)] #[pyclass( from_py_object, frozen, name = "JoinConstraint", module = "datafusion.expr" )] pub struct PyJoinConstraint { join_constraint: JoinConstraint, } impl From for PyJoinConstraint { fn from(join_constraint: JoinConstraint) -> PyJoinConstraint { PyJoinConstraint { join_constraint } } } impl From for JoinConstraint { fn from(join_constraint: PyJoinConstraint) -> Self { join_constraint.join_constraint } } #[pymethods] impl PyJoinConstraint { fn __repr__(&self) -> PyResult { match self.join_constraint { JoinConstraint::On => Ok("On".to_string()), JoinConstraint::Using => Ok("Using".to_string()), } } } #[pyclass( from_py_object, frozen, name = "Join", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyJoin { join: Join, } impl From for PyJoin { fn from(join: Join) -> PyJoin { PyJoin { join } } } impl From for Join { fn from(join: PyJoin) -> Self { join.join } } impl Display for PyJoin { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Join Left: {:?} Right: {:?} On: {:?} Filter: {:?} JoinType: {:?} JoinConstraint: {:?} Schema: {:?} NullEquality: {:?}", &self.join.left, &self.join.right, &self.join.on, &self.join.filter, &self.join.join_type, &self.join.join_constraint, &self.join.schema, &self.join.null_equality, ) } } #[pymethods] impl PyJoin { /// Retrieves the left input `LogicalPlan` to this `Join` node fn left(&self) -> PyResult { Ok(self.join.left.as_ref().clone().into()) } /// Retrieves the right input `LogicalPlan` to this `Join` node fn right(&self) -> PyResult { Ok(self.join.right.as_ref().clone().into()) } /// Retrieves the right input `LogicalPlan` to this `Join` node fn on(&self) -> PyResult> { Ok(self .join .on .iter() .map(|(l, r)| (PyExpr::from(l.clone()), PyExpr::from(r.clone()))) .collect()) } /// Retrieves the filter `Option` of this `Join` node fn filter(&self) -> PyResult> { Ok(self.join.filter.clone().map(Into::into)) } /// Retrieves the `JoinType` to this `Join` node fn join_type(&self) -> PyResult { Ok(self.join.join_type.into()) } /// Retrieves the `JoinConstraint` to this `Join` node fn join_constraint(&self) -> PyResult { Ok(self.join.join_constraint.into()) } /// Resulting Schema for this `Join` node instance fn schema(&self) -> PyResult { Ok(self.join.schema.as_ref().clone().into()) } /// If null_equals_null is true, null == null else null != null fn null_equals_null(&self) -> PyResult { match self.join.null_equality { NullEquality::NullEqualsNothing => Ok(false), NullEquality::NullEqualsNull => Ok(true), } } fn __repr__(&self) -> PyResult { Ok(format!("Join({self})")) } fn __name__(&self) -> PyResult { Ok("Join".to_string()) } } impl LogicalNode for PyJoin { fn inputs(&self) -> Vec { vec![ PyLogicalPlan::from((*self.join.left).clone()), PyLogicalPlan::from((*self.join.right).clone()), ] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/lambda.rs000066400000000000000000000037141521727353300242540ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::Lambda; use pyo3::prelude::*; use super::PyExpr; #[pyclass( from_py_object, frozen, name = "Lambda", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyLambda { lambda: Lambda, } impl From for PyLambda { fn from(lambda: Lambda) -> PyLambda { PyLambda { lambda } } } impl From for Lambda { fn from(lambda: PyLambda) -> Self { lambda.lambda } } impl Display for PyLambda { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Lambda(params={:?}, body={:?})", &self.lambda.params, &self.lambda.body, ) } } #[pymethods] impl PyLambda { /// Parameter names of the lambda. fn params(&self) -> Vec { self.lambda.params.clone() } /// Body expression of the lambda. fn body(&self) -> PyExpr { (*self.lambda.body).clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("Lambda({self})")) } fn __name__(&self) -> PyResult { Ok("Lambda".to_string()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/lambda_variable.rs000066400000000000000000000036001521727353300261130ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::LambdaVariable; use pyo3::prelude::*; #[pyclass( from_py_object, frozen, name = "LambdaVariable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyLambdaVariable { variable: LambdaVariable, } impl From for PyLambdaVariable { fn from(variable: LambdaVariable) -> PyLambdaVariable { PyLambdaVariable { variable } } } impl From for LambdaVariable { fn from(variable: PyLambdaVariable) -> Self { variable.variable } } impl Display for PyLambdaVariable { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!(f, "LambdaVariable({})", &self.variable.name) } } #[pymethods] impl PyLambdaVariable { /// Reference name of the lambda parameter. fn name(&self) -> String { self.variable.name.clone() } fn __repr__(&self) -> PyResult { Ok(format!("LambdaVariable({self})")) } fn __name__(&self) -> PyResult { Ok("LambdaVariable".to_string()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/like.rs000066400000000000000000000107471521727353300237640ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::Like; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "Like", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyLike { like: Like, } impl From for PyLike { fn from(like: Like) -> PyLike { PyLike { like } } } impl From for Like { fn from(like: PyLike) -> Self { like.like } } impl Display for PyLike { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Like Negated: {:?} Expr: {:?} Pattern: {:?} Escape_Char: {:?}", &self.negated(), &self.expr(), &self.pattern(), &self.escape_char() ) } } #[pymethods] impl PyLike { fn negated(&self) -> PyResult { Ok(self.like.negated) } fn expr(&self) -> PyResult { Ok((*self.like.expr).clone().into()) } fn pattern(&self) -> PyResult { Ok((*self.like.pattern).clone().into()) } fn escape_char(&self) -> PyResult> { Ok(self.like.escape_char) } fn __repr__(&self) -> String { format!("Like({self})") } } #[pyclass( from_py_object, frozen, name = "ILike", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyILike { like: Like, } impl From for PyILike { fn from(like: Like) -> PyILike { PyILike { like } } } impl From for Like { fn from(like: PyILike) -> Self { like.like } } impl Display for PyILike { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "ILike Negated: {:?} Expr: {:?} Pattern: {:?} Escape_Char: {:?}", &self.negated(), &self.expr(), &self.pattern(), &self.escape_char() ) } } #[pymethods] impl PyILike { fn negated(&self) -> PyResult { Ok(self.like.negated) } fn expr(&self) -> PyResult { Ok((*self.like.expr).clone().into()) } fn pattern(&self) -> PyResult { Ok((*self.like.pattern).clone().into()) } fn escape_char(&self) -> PyResult> { Ok(self.like.escape_char) } fn __repr__(&self) -> String { format!("Like({self})") } } #[pyclass( from_py_object, frozen, name = "SimilarTo", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySimilarTo { like: Like, } impl From for PySimilarTo { fn from(like: Like) -> PySimilarTo { PySimilarTo { like } } } impl From for Like { fn from(like: PySimilarTo) -> Self { like.like } } impl Display for PySimilarTo { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "SimilarTo Negated: {:?} Expr: {:?} Pattern: {:?} Escape_Char: {:?}", &self.negated(), &self.expr(), &self.pattern(), &self.escape_char() ) } } #[pymethods] impl PySimilarTo { fn negated(&self) -> PyResult { Ok(self.like.negated) } fn expr(&self) -> PyResult { Ok((*self.like.expr).clone().into()) } fn pattern(&self) -> PyResult { Ok((*self.like.pattern).clone().into()) } fn escape_char(&self) -> PyResult> { Ok(self.like.escape_char) } fn __repr__(&self) -> String { format!("Like({self})") } } apache-datafusion-python-54.0.0/crates/core/src/expr/limit.rs000066400000000000000000000055051521727353300241520ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::Limit; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Limit", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyLimit { limit: Limit, } impl From for PyLimit { fn from(limit: Limit) -> PyLimit { PyLimit { limit } } } impl From for Limit { fn from(limit: PyLimit) -> Self { limit.limit } } impl Display for PyLimit { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Limit Skip: {:?} Fetch: {:?} Input: {:?}", &self.limit.skip, &self.limit.fetch, &self.limit.input ) } } #[pymethods] impl PyLimit { // NOTE: Upstream now has expressions for skip and fetch // TODO: Do we still want to expose these? // REF: https://github.com/apache/datafusion/pull/12836 // /// Retrieves the skip value for this `Limit` // fn skip(&self) -> usize { // self.limit.skip // } // /// Retrieves the fetch value for this `Limit` // fn fetch(&self) -> Option { // self.limit.fetch // } /// Retrieves the input `LogicalPlan` to this `Limit` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Limit` node instance fn schema(&self) -> PyResult { Ok(self.limit.input.schema().as_ref().clone().into()) } fn __repr__(&self) -> PyResult { Ok(format!("Limit({self})")) } } impl LogicalNode for PyLimit { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.limit.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/literal.rs000066400000000000000000000122461521727353300244700ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::common::ScalarValue; use datafusion::logical_expr::expr::FieldMetadata; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::errors::PyDataFusionError; #[pyclass( from_py_object, name = "Literal", module = "datafusion.expr", subclass, frozen )] #[derive(Clone)] pub struct PyLiteral { pub value: ScalarValue, pub metadata: Option, } impl PyLiteral { pub fn new_with_metadata(value: ScalarValue, metadata: Option) -> PyLiteral { Self { value, metadata } } } impl From for ScalarValue { fn from(lit: PyLiteral) -> ScalarValue { lit.value } } impl From for PyLiteral { fn from(value: ScalarValue) -> PyLiteral { PyLiteral { value, metadata: None, } } } macro_rules! extract_scalar_value { ($self: expr, $variant: ident) => { match &$self.value { ScalarValue::$variant(value) => Ok(*value), other => Err(unexpected_literal_value(other)), } }; } #[pymethods] impl PyLiteral { /// Get the data type of this literal value fn data_type(&self) -> String { format!("{}", self.value.data_type()) } pub fn value_f32(&self) -> PyResult> { extract_scalar_value!(self, Float32) } pub fn value_f64(&self) -> PyResult> { extract_scalar_value!(self, Float64) } pub fn value_decimal128(&self) -> PyResult<(Option, u8, i8)> { match &self.value { ScalarValue::Decimal128(value, precision, scale) => Ok((*value, *precision, *scale)), other => Err(unexpected_literal_value(other)), } } pub fn value_i8(&self) -> PyResult> { extract_scalar_value!(self, Int8) } pub fn value_i16(&self) -> PyResult> { extract_scalar_value!(self, Int16) } pub fn value_i32(&self) -> PyResult> { extract_scalar_value!(self, Int32) } pub fn value_i64(&self) -> PyResult> { extract_scalar_value!(self, Int64) } pub fn value_u8(&self) -> PyResult> { extract_scalar_value!(self, UInt8) } pub fn value_u16(&self) -> PyResult> { extract_scalar_value!(self, UInt16) } pub fn value_u32(&self) -> PyResult> { extract_scalar_value!(self, UInt32) } pub fn value_u64(&self) -> PyResult> { extract_scalar_value!(self, UInt64) } pub fn value_date32(&self) -> PyResult> { extract_scalar_value!(self, Date32) } pub fn value_date64(&self) -> PyResult> { extract_scalar_value!(self, Date64) } pub fn value_time64(&self) -> PyResult> { extract_scalar_value!(self, Time64Nanosecond) } pub fn value_timestamp(&self) -> PyResult<(Option, Option)> { match &self.value { ScalarValue::TimestampNanosecond(iv, tz) | ScalarValue::TimestampMicrosecond(iv, tz) | ScalarValue::TimestampMillisecond(iv, tz) | ScalarValue::TimestampSecond(iv, tz) => { Ok((*iv, tz.as_ref().map(|s| s.as_ref().to_string()))) } other => Err(unexpected_literal_value(other)), } } pub fn value_bool(&self) -> PyResult> { extract_scalar_value!(self, Boolean) } pub fn value_string(&self) -> PyResult> { match &self.value { ScalarValue::Utf8(value) => Ok(value.clone()), other => Err(unexpected_literal_value(other)), } } pub fn value_interval_day_time(&self) -> PyResult> { match &self.value { ScalarValue::IntervalDayTime(Some(iv)) => Ok(Some((iv.days, iv.milliseconds))), ScalarValue::IntervalDayTime(None) => Ok(None), other => Err(unexpected_literal_value(other)), } } #[allow(clippy::wrong_self_convention)] fn into_type<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } fn __repr__(&self) -> PyResult { Ok(format!("{}", self.value)) } } fn unexpected_literal_value(value: &ScalarValue) -> PyErr { PyDataFusionError::Common(format!("getValue() - Unexpected value: {value}")).into() } apache-datafusion-python-54.0.0/crates/core/src/expr/logical_node.rs000066400000000000000000000023061521727353300254470ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use pyo3::{Bound, PyAny, PyResult, Python}; use crate::sql::logical::PyLogicalPlan; /// Representation of a `LogicalNode` in the in overall `LogicalPlan` /// any "node" shares these common traits in common. pub trait LogicalNode { /// The input plan to the current logical node instance. fn inputs(&self) -> Vec; fn to_variant<'py>(&self, py: Python<'py>) -> PyResult>; } apache-datafusion-python-54.0.0/crates/core/src/expr/placeholder.rs000066400000000000000000000033431521727353300253140ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use arrow::datatypes::Field; use arrow::pyarrow::PyArrowType; use datafusion::logical_expr::expr::Placeholder; use pyo3::prelude::*; use crate::common::data_type::PyDataType; #[pyclass( from_py_object, frozen, name = "Placeholder", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyPlaceholder { placeholder: Placeholder, } impl From for PyPlaceholder { fn from(placeholder: Placeholder) -> Self { PyPlaceholder { placeholder } } } #[pymethods] impl PyPlaceholder { fn id(&self) -> String { self.placeholder.id.clone() } fn data_type(&self) -> Option { self.placeholder .field .as_ref() .map(|f| f.data_type().clone().into()) } fn field(&self) -> Option> { self.placeholder .field .as_ref() .map(|f| f.as_ref().clone().into()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/projection.rs000066400000000000000000000071301521727353300252040ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::Expr; use datafusion::logical_expr::logical_plan::Projection; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::PyExpr; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Projection", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyProjection { pub projection: Projection, } impl PyProjection { pub fn new(projection: Projection) -> Self { Self { projection } } } impl From for PyProjection { fn from(projection: Projection) -> PyProjection { PyProjection { projection } } } impl From for Projection { fn from(proj: PyProjection) -> Self { proj.projection } } impl Display for PyProjection { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Projection \nExpr(s): {:?} \nInput: {:?} \nProjected Schema: {:?}", &self.projection.expr, &self.projection.input, &self.projection.schema, ) } } #[pymethods] impl PyProjection { /// Retrieves the expressions for this `Projection` fn projections(&self) -> PyResult> { Ok(self .projection .expr .iter() .map(|e| PyExpr::from(e.clone())) .collect()) } /// Retrieves the input `LogicalPlan` to this `Projection` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Projection` node instance fn schema(&self) -> PyResult { Ok((*self.projection.schema).clone().into()) } fn __repr__(&self) -> PyResult { Ok(format!("Projection({self})")) } fn __name__(&self) -> PyResult { Ok("Projection".to_string()) } } impl PyProjection { /// Projection: Gets the names of the fields that should be projected pub fn projected_expressions(local_expr: &PyExpr) -> Vec { let mut projs: Vec = Vec::new(); match &local_expr.expr { Expr::Alias(alias) => { let py_expr: PyExpr = PyExpr::from(*alias.expr.clone()); projs.extend_from_slice(Self::projected_expressions(&py_expr).as_slice()); } _ => projs.push(local_expr.clone()), } projs } } impl LogicalNode for PyProjection { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.projection.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/recursive_query.rs000066400000000000000000000062101521727353300262620ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::RecursiveQuery; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "RecursiveQuery", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyRecursiveQuery { query: RecursiveQuery, } impl From for RecursiveQuery { fn from(query: PyRecursiveQuery) -> Self { query.query } } impl From for PyRecursiveQuery { fn from(query: RecursiveQuery) -> PyRecursiveQuery { PyRecursiveQuery { query } } } impl Display for PyRecursiveQuery { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "RecursiveQuery {name:?} is_distinct:={is_distinct}", name = self.query.name, is_distinct = self.query.is_distinct ) } } #[pymethods] impl PyRecursiveQuery { #[new] fn new( name: String, static_term: PyLogicalPlan, recursive_term: PyLogicalPlan, is_distinct: bool, ) -> Self { Self { query: RecursiveQuery { name, static_term: static_term.plan(), recursive_term: recursive_term.plan(), is_distinct, }, } } fn name(&self) -> PyResult { Ok(self.query.name.clone()) } fn static_term(&self) -> PyLogicalPlan { PyLogicalPlan::from((*self.query.static_term).clone()) } fn recursive_term(&self) -> PyLogicalPlan { PyLogicalPlan::from((*self.query.recursive_term).clone()) } fn is_distinct(&self) -> PyResult { Ok(self.query.is_distinct) } fn __repr__(&self) -> PyResult { Ok(format!("RecursiveQuery({self})")) } fn __name__(&self) -> PyResult { Ok("RecursiveQuery".to_string()) } } impl LogicalNode for PyRecursiveQuery { fn inputs(&self) -> Vec { vec![ PyLogicalPlan::from((*self.query.static_term).clone()), PyLogicalPlan::from((*self.query.recursive_term).clone()), ] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/repartition.rs000066400000000000000000000077621521727353300254030ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::Repartition; use datafusion::logical_expr::{Expr, Partitioning}; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::PyExpr; use super::logical_node::LogicalNode; use crate::errors::py_type_err; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Repartition", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyRepartition { repartition: Repartition, } #[pyclass( from_py_object, frozen, name = "Partitioning", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyPartitioning { partitioning: Partitioning, } impl From for Partitioning { fn from(partitioning: PyPartitioning) -> Self { partitioning.partitioning } } impl From for PyPartitioning { fn from(partitioning: Partitioning) -> Self { PyPartitioning { partitioning } } } impl From for Repartition { fn from(repartition: PyRepartition) -> Self { repartition.repartition } } impl From for PyRepartition { fn from(repartition: Repartition) -> PyRepartition { PyRepartition { repartition } } } impl Display for PyRepartition { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Repartition input: {:?} partitioning_scheme: {:?}", &self.repartition.input, &self.repartition.partitioning_scheme, ) } } #[pymethods] impl PyRepartition { fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } fn partitioning_scheme(&self) -> PyResult { Ok(PyPartitioning { partitioning: self.repartition.partitioning_scheme.clone(), }) } fn distribute_list(&self) -> PyResult> { match &self.repartition.partitioning_scheme { Partitioning::DistributeBy(distribute_list) => Ok(distribute_list .iter() .map(|e| PyExpr::from(e.clone())) .collect()), _ => Err(py_type_err("unexpected repartition strategy")), } } fn distribute_columns(&self) -> PyResult { match &self.repartition.partitioning_scheme { Partitioning::DistributeBy(distribute_list) => Ok(distribute_list .iter() .map(|e| match &e { Expr::Column(column) => column.name.clone(), _ => panic!("Encountered a type other than Expr::Column"), }) .collect()), _ => Err(py_type_err("unexpected repartition strategy")), } } fn __repr__(&self) -> PyResult { Ok(format!("Repartition({self})")) } fn __name__(&self) -> PyResult { Ok("Repartition".to_string()) } } impl LogicalNode for PyRepartition { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.repartition.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/scalar_subquery.rs000066400000000000000000000027161521727353300262410ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::Subquery; use pyo3::prelude::*; use super::subquery::PySubquery; #[pyclass( from_py_object, frozen, name = "ScalarSubquery", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyScalarSubquery { subquery: Subquery, } impl From for Subquery { fn from(subquery: PyScalarSubquery) -> Self { subquery.subquery } } impl From for PyScalarSubquery { fn from(subquery: Subquery) -> PyScalarSubquery { PyScalarSubquery { subquery } } } #[pymethods] impl PyScalarSubquery { fn subquery(&self) -> PySubquery { self.subquery.clone().into() } } apache-datafusion-python-54.0.0/crates/core/src/expr/scalar_variable.rs000066400000000000000000000032541521727353300261450ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use arrow::datatypes::FieldRef; use pyo3::prelude::*; use crate::common::data_type::PyDataType; #[pyclass( from_py_object, frozen, name = "ScalarVariable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyScalarVariable { field: FieldRef, variables: Vec, } impl PyScalarVariable { pub fn new(field: &FieldRef, variables: &[String]) -> Self { Self { field: field.to_owned(), variables: variables.to_vec(), } } } #[pymethods] impl PyScalarVariable { /// Get the data type fn data_type(&self) -> PyResult { Ok(self.field.data_type().clone().into()) } fn variables(&self) -> PyResult> { Ok(self.variables.clone()) } fn __repr__(&self) -> PyResult { Ok(format!("{}{:?}", self.field.data_type(), self.variables)) } } apache-datafusion-python-54.0.0/crates/core/src/expr/set_comparison.rs000066400000000000000000000032501521727353300260540ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::expr::SetComparison; use pyo3::prelude::*; use super::subquery::PySubquery; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "SetComparison", module = "datafusion.set_comparison", subclass )] #[derive(Clone)] pub struct PySetComparison { set_comparison: SetComparison, } impl From for PySetComparison { fn from(set_comparison: SetComparison) -> Self { PySetComparison { set_comparison } } } #[pymethods] impl PySetComparison { fn expr(&self) -> PyExpr { (*self.set_comparison.expr).clone().into() } fn subquery(&self) -> PySubquery { self.set_comparison.subquery.clone().into() } fn op(&self) -> String { format!("{}", self.set_comparison.op) } fn quantifier(&self) -> String { format!("{}", self.set_comparison.quantifier) } } apache-datafusion-python-54.0.0/crates/core/src/expr/signature.rs000066400000000000000000000025011521727353300250260ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion::logical_expr::{TypeSignature, Volatility}; use pyo3::prelude::*; #[allow(dead_code)] #[pyclass( from_py_object, frozen, name = "Signature", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySignature { type_signature: TypeSignature, volatility: Volatility, } impl PySignature { pub fn new(type_signature: TypeSignature, volatility: Volatility) -> Self { Self { type_signature, volatility, } } } #[pymethods] impl PySignature {} apache-datafusion-python-54.0.0/crates/core/src/expr/sort.rs000066400000000000000000000056161521727353300240260ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::common::DataFusionError; use datafusion::logical_expr::logical_plan::Sort; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::logical_node::LogicalNode; use crate::expr::sort_expr::PySortExpr; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Sort", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySort { sort: Sort, } impl From for PySort { fn from(sort: Sort) -> PySort { PySort { sort } } } impl TryFrom for Sort { type Error = DataFusionError; fn try_from(agg: PySort) -> Result { Ok(agg.sort) } } impl Display for PySort { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Sort \nExpr(s): {:?} \nInput: {:?} \nSchema: {:?}", &self.sort.expr, self.sort.input, self.sort.input.schema() ) } } #[pymethods] impl PySort { /// Retrieves the sort expressions for this `Sort` fn sort_exprs(&self) -> PyResult> { Ok(self .sort .expr .iter() .map(|e| PySortExpr::from(e.clone())) .collect()) } fn get_fetch_val(&self) -> PyResult> { Ok(self.sort.fetch) } /// Retrieves the input `LogicalPlan` to this `Sort` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Sort` node instance fn schema(&self) -> PyDFSchema { self.sort.input.schema().as_ref().clone().into() } fn __repr__(&self) -> PyResult { Ok(format!("Sort({self})")) } } impl LogicalNode for PySort { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.sort.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/sort_expr.rs000066400000000000000000000047131521727353300250610ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::SortExpr; use pyo3::prelude::*; use crate::expr::PyExpr; #[pyclass( from_py_object, frozen, name = "SortExpr", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySortExpr { pub(crate) sort: SortExpr, } impl From for SortExpr { fn from(sort: PySortExpr) -> Self { sort.sort } } impl From for PySortExpr { fn from(sort: SortExpr) -> PySortExpr { PySortExpr { sort } } } impl Display for PySortExpr { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Sort Expr: {:?} Asc: {:?} NullsFirst: {:?}", &self.sort.expr, &self.sort.asc, &self.sort.nulls_first ) } } pub fn to_sort_expressions(order_by: Vec) -> Vec { order_by.iter().map(|e| e.sort.clone()).collect() } pub fn py_sort_expr_list(expr: &[SortExpr]) -> PyResult> { Ok(expr.iter().map(|e| PySortExpr::from(e.clone())).collect()) } #[pymethods] impl PySortExpr { #[new] fn new(expr: PyExpr, asc: bool, nulls_first: bool) -> Self { Self { sort: SortExpr { expr: expr.into(), asc, nulls_first, }, } } fn expr(&self) -> PyResult { Ok(self.sort.expr.clone().into()) } fn ascending(&self) -> PyResult { Ok(self.sort.asc) } fn nulls_first(&self) -> PyResult { Ok(self.sort.nulls_first) } fn __repr__(&self) -> String { format!("{self}") } } apache-datafusion-python-54.0.0/crates/core/src/expr/statement.rs000066400000000000000000000334311521727353300250370ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use arrow::datatypes::Field; use arrow::pyarrow::PyArrowType; use datafusion::logical_expr::{ Deallocate, Execute, Prepare, ResetVariable, SetVariable, TransactionAccessMode, TransactionConclusion, TransactionEnd, TransactionIsolationLevel, TransactionStart, }; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::PyExpr; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "TransactionStart", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyTransactionStart { transaction_start: TransactionStart, } impl From for PyTransactionStart { fn from(transaction_start: TransactionStart) -> PyTransactionStart { PyTransactionStart { transaction_start } } } impl TryFrom for TransactionStart { type Error = PyErr; fn try_from(py: PyTransactionStart) -> Result { Ok(py.transaction_start) } } impl LogicalNode for PyTransactionStart { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "TransactionAccessMode", module = "datafusion.expr" )] pub enum PyTransactionAccessMode { ReadOnly, ReadWrite, } impl From for PyTransactionAccessMode { fn from(access_mode: TransactionAccessMode) -> PyTransactionAccessMode { match access_mode { TransactionAccessMode::ReadOnly => PyTransactionAccessMode::ReadOnly, TransactionAccessMode::ReadWrite => PyTransactionAccessMode::ReadWrite, } } } impl TryFrom for TransactionAccessMode { type Error = PyErr; fn try_from(py: PyTransactionAccessMode) -> Result { match py { PyTransactionAccessMode::ReadOnly => Ok(TransactionAccessMode::ReadOnly), PyTransactionAccessMode::ReadWrite => Ok(TransactionAccessMode::ReadWrite), } } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "TransactionIsolationLevel", module = "datafusion.expr" )] pub enum PyTransactionIsolationLevel { ReadUncommitted, ReadCommitted, RepeatableRead, Serializable, Snapshot, } impl From for PyTransactionIsolationLevel { fn from(isolation_level: TransactionIsolationLevel) -> PyTransactionIsolationLevel { match isolation_level { TransactionIsolationLevel::ReadUncommitted => { PyTransactionIsolationLevel::ReadUncommitted } TransactionIsolationLevel::ReadCommitted => PyTransactionIsolationLevel::ReadCommitted, TransactionIsolationLevel::RepeatableRead => { PyTransactionIsolationLevel::RepeatableRead } TransactionIsolationLevel::Serializable => PyTransactionIsolationLevel::Serializable, TransactionIsolationLevel::Snapshot => PyTransactionIsolationLevel::Snapshot, } } } impl TryFrom for TransactionIsolationLevel { type Error = PyErr; fn try_from(value: PyTransactionIsolationLevel) -> Result { match value { PyTransactionIsolationLevel::ReadUncommitted => { Ok(TransactionIsolationLevel::ReadUncommitted) } PyTransactionIsolationLevel::ReadCommitted => { Ok(TransactionIsolationLevel::ReadCommitted) } PyTransactionIsolationLevel::RepeatableRead => { Ok(TransactionIsolationLevel::RepeatableRead) } PyTransactionIsolationLevel::Serializable => { Ok(TransactionIsolationLevel::Serializable) } PyTransactionIsolationLevel::Snapshot => Ok(TransactionIsolationLevel::Snapshot), } } } #[pymethods] impl PyTransactionStart { #[new] pub fn new( access_mode: PyTransactionAccessMode, isolation_level: PyTransactionIsolationLevel, ) -> PyResult { let access_mode = access_mode.try_into()?; let isolation_level = isolation_level.try_into()?; Ok(PyTransactionStart { transaction_start: TransactionStart { access_mode, isolation_level, }, }) } pub fn access_mode(&self) -> PyResult { Ok(self.transaction_start.access_mode.clone().into()) } pub fn isolation_level(&self) -> PyResult { Ok(self.transaction_start.isolation_level.clone().into()) } } #[pyclass( from_py_object, frozen, name = "TransactionEnd", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyTransactionEnd { transaction_end: TransactionEnd, } impl From for PyTransactionEnd { fn from(transaction_end: TransactionEnd) -> PyTransactionEnd { PyTransactionEnd { transaction_end } } } impl TryFrom for TransactionEnd { type Error = PyErr; fn try_from(py: PyTransactionEnd) -> Result { Ok(py.transaction_end) } } impl LogicalNode for PyTransactionEnd { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord)] #[pyclass( from_py_object, frozen, eq, eq_int, name = "TransactionConclusion", module = "datafusion.expr" )] pub enum PyTransactionConclusion { Commit, Rollback, } impl From for PyTransactionConclusion { fn from(value: TransactionConclusion) -> Self { match value { TransactionConclusion::Commit => PyTransactionConclusion::Commit, TransactionConclusion::Rollback => PyTransactionConclusion::Rollback, } } } impl TryFrom for TransactionConclusion { type Error = PyErr; fn try_from(value: PyTransactionConclusion) -> Result { match value { PyTransactionConclusion::Commit => Ok(TransactionConclusion::Commit), PyTransactionConclusion::Rollback => Ok(TransactionConclusion::Rollback), } } } #[pymethods] impl PyTransactionEnd { #[new] pub fn new(conclusion: PyTransactionConclusion, chain: bool) -> PyResult { let conclusion = conclusion.try_into()?; Ok(PyTransactionEnd { transaction_end: TransactionEnd { conclusion, chain }, }) } pub fn conclusion(&self) -> PyResult { Ok(self.transaction_end.conclusion.clone().into()) } pub fn chain(&self) -> bool { self.transaction_end.chain } } #[pyclass( from_py_object, frozen, name = "ResetVariable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyResetVariable { reset_variable: ResetVariable, } impl From for PyResetVariable { fn from(reset_variable: ResetVariable) -> PyResetVariable { PyResetVariable { reset_variable } } } impl TryFrom for ResetVariable { type Error = PyErr; fn try_from(py: PyResetVariable) -> Result { Ok(py.reset_variable) } } impl LogicalNode for PyResetVariable { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyResetVariable { #[new] pub fn new(variable: String) -> Self { PyResetVariable { reset_variable: ResetVariable { variable }, } } pub fn variable(&self) -> String { self.reset_variable.variable.clone() } } #[pyclass( from_py_object, frozen, name = "SetVariable", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySetVariable { set_variable: SetVariable, } impl From for PySetVariable { fn from(set_variable: SetVariable) -> PySetVariable { PySetVariable { set_variable } } } impl TryFrom for SetVariable { type Error = PyErr; fn try_from(py: PySetVariable) -> Result { Ok(py.set_variable) } } impl LogicalNode for PySetVariable { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PySetVariable { #[new] pub fn new(variable: String, value: String) -> Self { PySetVariable { set_variable: SetVariable { variable, value }, } } pub fn variable(&self) -> String { self.set_variable.variable.clone() } pub fn value(&self) -> String { self.set_variable.value.clone() } } #[pyclass( from_py_object, frozen, name = "Prepare", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyPrepare { prepare: Prepare, } impl From for PyPrepare { fn from(prepare: Prepare) -> PyPrepare { PyPrepare { prepare } } } impl TryFrom for Prepare { type Error = PyErr; fn try_from(py: PyPrepare) -> Result { Ok(py.prepare) } } impl LogicalNode for PyPrepare { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.prepare.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyPrepare { #[new] pub fn new(name: String, fields: Vec>, input: PyLogicalPlan) -> Self { let input = input.plan().clone(); let fields = fields.into_iter().map(|field| Arc::new(field.0)).collect(); PyPrepare { prepare: Prepare { name, fields, input, }, } } pub fn name(&self) -> String { self.prepare.name.clone() } pub fn fields(&self) -> Vec> { self.prepare .fields .clone() .into_iter() .map(|f| f.as_ref().clone().into()) .collect() } pub fn input(&self) -> PyLogicalPlan { PyLogicalPlan { plan: self.prepare.input.clone(), } } } #[pyclass( from_py_object, frozen, name = "Execute", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyExecute { execute: Execute, } impl From for PyExecute { fn from(execute: Execute) -> PyExecute { PyExecute { execute } } } impl TryFrom for Execute { type Error = PyErr; fn try_from(py: PyExecute) -> Result { Ok(py.execute) } } impl LogicalNode for PyExecute { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyExecute { #[new] pub fn new(name: String, parameters: Vec) -> Self { let parameters = parameters .into_iter() .map(|parameter| parameter.into()) .collect(); PyExecute { execute: Execute { name, parameters }, } } pub fn name(&self) -> String { self.execute.name.clone() } pub fn parameters(&self) -> Vec { self.execute .parameters .clone() .into_iter() .map(|t| t.into()) .collect() } } #[pyclass( from_py_object, frozen, name = "Deallocate", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyDeallocate { deallocate: Deallocate, } impl From for PyDeallocate { fn from(deallocate: Deallocate) -> PyDeallocate { PyDeallocate { deallocate } } } impl TryFrom for Deallocate { type Error = PyErr; fn try_from(py: PyDeallocate) -> Result { Ok(py.deallocate) } } impl LogicalNode for PyDeallocate { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyDeallocate { #[new] pub fn new(name: String) -> Self { PyDeallocate { deallocate: Deallocate { name }, } } pub fn name(&self) -> String { self.deallocate.name.clone() } } apache-datafusion-python-54.0.0/crates/core/src/expr/subquery.rs000066400000000000000000000044341521727353300247130ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::Subquery; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Subquery", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySubquery { subquery: Subquery, } impl From for Subquery { fn from(subquery: PySubquery) -> Self { subquery.subquery } } impl From for PySubquery { fn from(subquery: Subquery) -> PySubquery { PySubquery { subquery } } } impl Display for PySubquery { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Subquery Subquery: {:?} outer_ref_columns: {:?}", self.subquery.subquery, self.subquery.outer_ref_columns, ) } } #[pymethods] impl PySubquery { /// Retrieves the input `LogicalPlan` to this `Projection` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } fn __repr__(&self) -> PyResult { Ok(format!("Subquery({self})")) } fn __name__(&self) -> PyResult { Ok("Subquery".to_string()) } } impl LogicalNode for PySubquery { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/subquery_alias.rs000066400000000000000000000054631521727353300260670ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::SubqueryAlias; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "SubqueryAlias", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PySubqueryAlias { subquery_alias: SubqueryAlias, } impl From for SubqueryAlias { fn from(subquery_alias: PySubqueryAlias) -> Self { subquery_alias.subquery_alias } } impl From for PySubqueryAlias { fn from(subquery_alias: SubqueryAlias) -> PySubqueryAlias { PySubqueryAlias { subquery_alias } } } impl Display for PySubqueryAlias { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "SubqueryAlias Inputs(s): {:?} Alias: {:?} Schema: {:?}", self.subquery_alias.input, self.subquery_alias.alias, self.subquery_alias.schema, ) } } #[pymethods] impl PySubqueryAlias { /// Retrieves the input `LogicalPlan` to this `Projection` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Projection` node instance fn schema(&self) -> PyResult { Ok((*self.subquery_alias.schema).clone().into()) } fn alias(&self) -> PyResult { Ok(self.subquery_alias.alias.to_string()) } fn __repr__(&self) -> PyResult { Ok(format!("SubqueryAlias({self})")) } fn __name__(&self) -> PyResult { Ok("SubqueryAlias".to_string()) } } impl LogicalNode for PySubqueryAlias { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.subquery_alias.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/table_scan.rs000066400000000000000000000117161521727353300251300ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::common::TableReference; use datafusion::logical_expr::logical_plan::TableScan; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::PyExpr; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "TableScan", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyTableScan { table_scan: TableScan, } impl PyTableScan { pub fn new(table_scan: TableScan) -> Self { Self { table_scan } } } impl From for TableScan { fn from(tbl_scan: PyTableScan) -> TableScan { tbl_scan.table_scan } } impl From for PyTableScan { fn from(table_scan: TableScan) -> PyTableScan { PyTableScan { table_scan } } } impl Display for PyTableScan { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "TableScan\nTable Name: {} Projections: {:?} Projected Schema: {:?} Filters: {:?}", &self.table_scan.table_name, &self.py_projections(), &self.py_schema(), &self.py_filters(), ) } } #[pymethods] impl PyTableScan { /// Retrieves the name of the table represented by this `TableScan` instance #[pyo3(name = "table_name")] fn py_table_name(&self) -> PyResult { Ok(format!("{}", self.table_scan.table_name)) } #[pyo3(name = "fqn")] fn fqn(&self) -> PyResult<(Option, Option, String)> { let table_ref: TableReference = self.table_scan.table_name.clone(); Ok(match table_ref { TableReference::Bare { table } => (None, None, table.to_string()), TableReference::Partial { schema, table } => { (None, Some(schema.to_string()), table.to_string()) } TableReference::Full { catalog, schema, table, } => ( Some(catalog.to_string()), Some(schema.to_string()), table.to_string(), ), }) } /// The column indexes that should be. Note if this is empty then /// all columns should be read by the `TableProvider`. This function /// provides a Tuple of the (index, column_name) to make things simpler /// for the calling code since often times the name is preferred to /// the index which is a lower level abstraction. #[pyo3(name = "projection")] fn py_projections(&self) -> PyResult> { match &self.table_scan.projection { Some(indices) => { let schema = self.table_scan.source.schema(); Ok(indices .iter() .map(|i| (*i, schema.field(*i).name().to_string())) .collect()) } None => Ok(vec![]), } } /// Resulting schema from the `TableScan` operation #[pyo3(name = "schema")] fn py_schema(&self) -> PyResult { Ok((*self.table_scan.projected_schema).clone().into()) } /// Certain `TableProvider` physical readers offer the capability to filter rows that /// are read at read time. These `filters` are contained here. #[pyo3(name = "filters")] fn py_filters(&self) -> PyResult> { Ok(self .table_scan .filters .iter() .map(|expr| PyExpr::from(expr.clone())) .collect()) } /// Optional number of rows that should be read at read time by the `TableProvider` #[pyo3(name = "fetch")] fn py_fetch(&self) -> PyResult> { Ok(self.table_scan.fetch) } fn __repr__(&self) -> PyResult { Ok(format!("TableScan({self})")) } } impl LogicalNode for PyTableScan { fn inputs(&self) -> Vec { // table scans are leaf nodes and do not have inputs vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/union.rs000066400000000000000000000050201521727353300241540ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::Union; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Union", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyUnion { union_: Union, } impl From for PyUnion { fn from(union_: Union) -> PyUnion { PyUnion { union_ } } } impl From for Union { fn from(union_: PyUnion) -> Self { union_.union_ } } impl Display for PyUnion { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Union Inputs: {:?} Schema: {:?}", &self.union_.inputs, &self.union_.schema, ) } } #[pymethods] impl PyUnion { /// Retrieves the input `LogicalPlan` to this `Union` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Union` node instance fn schema(&self) -> PyResult { Ok(self.union_.schema.as_ref().clone().into()) } fn __repr__(&self) -> PyResult { Ok(format!("Union({self})")) } fn __name__(&self) -> PyResult { Ok("Union".to_string()) } } impl LogicalNode for PyUnion { fn inputs(&self) -> Vec { self.union_ .inputs .iter() .map(|x| x.as_ref().clone().into()) .collect() } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/unnest.rs000066400000000000000000000047521521727353300243530ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::logical_plan::Unnest; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::common::df_schema::PyDFSchema; use crate::expr::logical_node::LogicalNode; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Unnest", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyUnnest { unnest_: Unnest, } impl From for PyUnnest { fn from(unnest_: Unnest) -> PyUnnest { PyUnnest { unnest_ } } } impl From for Unnest { fn from(unnest_: PyUnnest) -> Self { unnest_.unnest_ } } impl Display for PyUnnest { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Unnest Inputs: {:?} Schema: {:?}", &self.unnest_.input, &self.unnest_.schema, ) } } #[pymethods] impl PyUnnest { /// Retrieves the input `LogicalPlan` to this `Unnest` node fn input(&self) -> PyResult> { Ok(Self::inputs(self)) } /// Resulting Schema for this `Unnest` node instance fn schema(&self) -> PyResult { Ok(self.unnest_.schema.as_ref().clone().into()) } fn __repr__(&self) -> PyResult { Ok(format!("Unnest({self})")) } fn __name__(&self) -> PyResult { Ok("Unnest".to_string()) } } impl LogicalNode for PyUnnest { fn inputs(&self) -> Vec { vec![PyLogicalPlan::from((*self.unnest_.input).clone())] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/expr/unnest_expr.rs000066400000000000000000000036121521727353300254030ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::logical_expr::expr::Unnest; use pyo3::prelude::*; use super::PyExpr; #[pyclass( from_py_object, frozen, name = "UnnestExpr", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyUnnestExpr { unnest: Unnest, } impl From for PyUnnestExpr { fn from(unnest: Unnest) -> PyUnnestExpr { PyUnnestExpr { unnest } } } impl From for Unnest { fn from(unnest: PyUnnestExpr) -> Self { unnest.unnest } } impl Display for PyUnnestExpr { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Unnest Expr: {:?}", &self.unnest.expr, ) } } #[pymethods] impl PyUnnestExpr { /// Retrieves the expression that is being unnested fn expr(&self) -> PyResult { Ok((*self.unnest.expr).clone().into()) } fn __repr__(&self) -> PyResult { Ok(format!("UnnestExpr({self})")) } fn __name__(&self) -> PyResult { Ok("UnnestExpr".to_string()) } } apache-datafusion-python-54.0.0/crates/core/src/expr/values.rs000066400000000000000000000047431521727353300243360ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::logical_expr::Values; use pyo3::prelude::*; use pyo3::{IntoPyObjectExt, PyErr, PyResult, Python, pyclass}; use super::PyExpr; use super::logical_node::LogicalNode; use crate::common::df_schema::PyDFSchema; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Values", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyValues { values: Values, } impl From for PyValues { fn from(values: Values) -> PyValues { PyValues { values } } } impl TryFrom for Values { type Error = PyErr; fn try_from(py: PyValues) -> Result { Ok(py.values) } } impl LogicalNode for PyValues { fn inputs(&self) -> Vec { vec![] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } #[pymethods] impl PyValues { #[new] pub fn new(schema: PyDFSchema, values: Vec>) -> PyResult { let values = values .into_iter() .map(|row| row.into_iter().map(|expr| expr.into()).collect()) .collect(); Ok(PyValues { values: Values { schema: Arc::new(schema.into()), values, }, }) } pub fn schema(&self) -> PyResult { Ok((*self.values.schema).clone().into()) } pub fn values(&self) -> Vec> { self.values .values .clone() .into_iter() .map(|row| row.into_iter().map(|expr| expr.into()).collect()) .collect() } } apache-datafusion-python-54.0.0/crates/core/src/expr/window.rs000066400000000000000000000235461521727353300243500ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{self, Display, Formatter}; use datafusion::common::{DataFusionError, ScalarValue}; use datafusion::logical_expr::{Expr, Window, WindowFrame, WindowFrameBound, WindowFrameUnits}; use pyo3::IntoPyObjectExt; use pyo3::exceptions::PyNotImplementedError; use pyo3::prelude::*; use super::py_expr_list; use crate::common::data_type::PyScalarValue; use crate::common::df_schema::PyDFSchema; use crate::errors::{PyDataFusionResult, py_type_err}; use crate::expr::PyExpr; use crate::expr::logical_node::LogicalNode; use crate::expr::sort_expr::{PySortExpr, py_sort_expr_list}; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "WindowExpr", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyWindowExpr { window: Window, } #[pyclass( from_py_object, frozen, name = "WindowFrame", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyWindowFrame { window_frame: WindowFrame, } impl From for WindowFrame { fn from(window_frame: PyWindowFrame) -> Self { window_frame.window_frame } } impl From for PyWindowFrame { fn from(window_frame: WindowFrame) -> PyWindowFrame { PyWindowFrame { window_frame } } } #[pyclass( from_py_object, frozen, name = "WindowFrameBound", module = "datafusion.expr", subclass )] #[derive(Clone)] pub struct PyWindowFrameBound { frame_bound: WindowFrameBound, } impl From for Window { fn from(window: PyWindowExpr) -> Window { window.window } } impl From for PyWindowExpr { fn from(window: Window) -> PyWindowExpr { PyWindowExpr { window } } } impl From for PyWindowFrameBound { fn from(frame_bound: WindowFrameBound) -> Self { PyWindowFrameBound { frame_bound } } } impl Display for PyWindowExpr { fn fmt(&self, f: &mut Formatter) -> fmt::Result { write!( f, "Over\n Window Expr: {:?} Schema: {:?}", &self.window.window_expr, &self.window.schema ) } } impl Display for PyWindowFrame { fn fmt(&self, f: &mut Formatter) -> std::fmt::Result { write!( f, "OVER ({} BETWEEN {} AND {})", self.window_frame.units, self.window_frame.start_bound, self.window_frame.end_bound ) } } #[pymethods] impl PyWindowExpr { /// Returns the schema of the Window pub fn schema(&self) -> PyResult { Ok(self.window.schema.as_ref().clone().into()) } /// Returns window expressions pub fn get_window_expr(&self) -> PyResult> { py_expr_list(&self.window.window_expr) } /// Returns order by columns in a window function expression pub fn get_sort_exprs(&self, expr: PyExpr) -> PyResult> { match expr.expr.unalias() { Expr::WindowFunction(boxed_window_fn) => { py_sort_expr_list(&boxed_window_fn.params.order_by) } other => Err(not_window_function_err(other)), } } /// Return partition by columns in a window function expression pub fn get_partition_exprs(&self, expr: PyExpr) -> PyResult> { match expr.expr.unalias() { Expr::WindowFunction(boxed_window_fn) => { py_expr_list(&boxed_window_fn.params.partition_by) } other => Err(not_window_function_err(other)), } } /// Return input args for window function pub fn get_args(&self, expr: PyExpr) -> PyResult> { match expr.expr.unalias() { Expr::WindowFunction(boxed_window_fn) => py_expr_list(&boxed_window_fn.params.args), other => Err(not_window_function_err(other)), } } /// Return window function name pub fn window_func_name(&self, expr: PyExpr) -> PyResult { match expr.expr.unalias() { Expr::WindowFunction(boxed_window_fn) => Ok(boxed_window_fn.fun.to_string()), other => Err(not_window_function_err(other)), } } /// Returns a Pywindow frame for a given window function expression pub fn get_frame(&self, expr: PyExpr) -> Option { match expr.expr.unalias() { Expr::WindowFunction(boxed_window_fn) => { Some(boxed_window_fn.params.window_frame.into()) } _ => None, } } } fn not_window_function_err(expr: Expr) -> PyErr { py_type_err(format!( "Provided {} Expr {:?} is not a WindowFunction type", expr.variant_name(), expr )) } #[pymethods] impl PyWindowFrame { #[new] #[pyo3(signature=(unit, start_bound, end_bound))] pub fn new( unit: &str, start_bound: Option, end_bound: Option, ) -> PyResult { let units = unit.to_ascii_lowercase(); let units = match units.as_str() { "rows" => WindowFrameUnits::Rows, "range" => WindowFrameUnits::Range, "groups" => WindowFrameUnits::Groups, _ => { return Err(PyNotImplementedError::new_err(format!("{units:?}"))); } }; let start_bound = match start_bound { Some(start_bound) => WindowFrameBound::Preceding(start_bound.0), None => match units { WindowFrameUnits::Range => WindowFrameBound::Preceding(ScalarValue::UInt64(None)), WindowFrameUnits::Rows => WindowFrameBound::Preceding(ScalarValue::UInt64(None)), WindowFrameUnits::Groups => { return Err(PyNotImplementedError::new_err(format!("{units:?}"))); } }, }; let end_bound = match end_bound { Some(end_bound) => WindowFrameBound::Following(end_bound.0), None => match units { WindowFrameUnits::Rows => WindowFrameBound::Following(ScalarValue::UInt64(None)), WindowFrameUnits::Range => WindowFrameBound::Following(ScalarValue::UInt64(None)), WindowFrameUnits::Groups => { return Err(PyNotImplementedError::new_err(format!("{units:?}"))); } }, }; Ok(PyWindowFrame { window_frame: WindowFrame::new_bounds(units, start_bound, end_bound), }) } /// Returns the window frame units for the bounds pub fn get_frame_units(&self) -> PyResult { Ok(self.window_frame.units.to_string()) } /// Returns starting bound pub fn get_lower_bound(&self) -> PyResult { Ok(self.window_frame.start_bound.clone().into()) } /// Returns end bound pub fn get_upper_bound(&self) -> PyResult { Ok(self.window_frame.end_bound.clone().into()) } /// Get a String representation of this window frame fn __repr__(&self) -> String { format!("{self}") } } #[pymethods] impl PyWindowFrameBound { /// Returns if the frame bound is current row pub fn is_current_row(&self) -> bool { matches!(self.frame_bound, WindowFrameBound::CurrentRow) } /// Returns if the frame bound is preceding pub fn is_preceding(&self) -> bool { matches!(self.frame_bound, WindowFrameBound::Preceding(_)) } /// Returns if the frame bound is following pub fn is_following(&self) -> bool { matches!(self.frame_bound, WindowFrameBound::Following(_)) } /// Returns the offset of the window frame pub fn get_offset(&self) -> PyDataFusionResult> { match &self.frame_bound { WindowFrameBound::Preceding(val) | WindowFrameBound::Following(val) => match val { x if x.is_null() => Ok(None), ScalarValue::UInt64(v) => Ok(*v), // The cast below is only safe because window bounds cannot be negative ScalarValue::Int64(v) => Ok(v.map(|n| n as u64)), ScalarValue::Utf8(Some(s)) => match s.parse::() { Ok(s) => Ok(Some(s)), Err(_e) => Err(DataFusionError::Plan(format!( "Unable to parse u64 from Utf8 value '{s}'" )) .into()), }, ref x => { Err(DataFusionError::Plan(format!("Unexpected window frame bound: {x}")).into()) } }, WindowFrameBound::CurrentRow => Ok(None), } } /// Returns if the frame bound is unbounded pub fn is_unbounded(&self) -> PyResult { match &self.frame_bound { WindowFrameBound::Preceding(v) | WindowFrameBound::Following(v) => Ok(v.is_null()), WindowFrameBound::CurrentRow => Ok(false), } } } impl LogicalNode for PyWindowExpr { fn inputs(&self) -> Vec { vec![self.window.input.as_ref().clone().into()] } fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { self.clone().into_bound_py_any(py) } } apache-datafusion-python-54.0.0/crates/core/src/functions.rs000066400000000000000000001263071521727353300240720ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use datafusion::common::{Column, ScalarValue, TableReference}; use datafusion::logical_expr::expr::{Alias, FieldMetadata, NullTreatment as DFNullTreatment}; use datafusion::logical_expr::{Expr, ExprFunctionExt, lit}; use datafusion::{functions, functions_aggregate, functions_window}; use pyo3::prelude::*; use pyo3::wrap_pyfunction; use crate::common::data_type::{NullTreatment, PyScalarValue}; use crate::errors::PyDataFusionResult; use crate::expr::PyExpr; use crate::expr::conditional_expr::PyCaseBuilder; use crate::expr::sort_expr::{PySortExpr, to_sort_expressions}; use crate::expr::window::PyWindowFrame; pub(crate) fn add_builder_fns_to_aggregate( agg_fn: Expr, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { // Since ExprFuncBuilder::new() is private, we can guarantee initializing // a builder with an `null_treatment` with option None let mut builder = agg_fn.null_treatment(None); if let Some(order_by_cols) = order_by { let order_by_cols = to_sort_expressions(order_by_cols); builder = builder.order_by(order_by_cols); } if let Some(true) = distinct { builder = builder.distinct(); } if let Some(filter) = filter { builder = builder.filter(filter.expr); } builder = builder.null_treatment(null_treatment.map(DFNullTreatment::from)); Ok(builder.build()?.into()) } #[pyfunction] fn in_list(expr: PyExpr, value: Vec, negated: bool) -> PyExpr { datafusion::logical_expr::in_list( expr.expr, value.into_iter().map(|x| x.expr).collect::>(), negated, ) .into() } #[pyfunction] fn make_array(exprs: Vec) -> PyExpr { datafusion::functions_nested::expr_fn::make_array(exprs.into_iter().map(|x| x.into()).collect()) .into() } #[pyfunction] fn array_concat(exprs: Vec) -> PyExpr { let exprs = exprs.into_iter().map(|x| x.into()).collect(); datafusion::functions_nested::expr_fn::array_concat(exprs).into() } #[pyfunction] fn array_cat(exprs: Vec) -> PyExpr { array_concat(exprs) } #[pyfunction] fn array_distance(array1: PyExpr, array2: PyExpr) -> PyExpr { let args = vec![array1.into(), array2.into()]; Expr::ScalarFunction(datafusion::logical_expr::expr::ScalarFunction::new_udf( datafusion::functions_nested::distance::array_distance_udf(), args, )) .into() } #[pyfunction] fn arrays_zip(exprs: Vec) -> PyExpr { let exprs = exprs.into_iter().map(|x| x.into()).collect(); datafusion::functions_nested::expr_fn::arrays_zip(exprs).into() } #[pyfunction] #[pyo3(signature = (string, delimiter, null_string=None))] fn string_to_array(string: PyExpr, delimiter: PyExpr, null_string: Option) -> PyExpr { let mut args = vec![string.into(), delimiter.into()]; if let Some(null_string) = null_string { args.push(null_string.into()); } Expr::ScalarFunction(datafusion::logical_expr::expr::ScalarFunction::new_udf( datafusion::functions_nested::string::string_to_array_udf(), args, )) .into() } #[pyfunction] #[pyo3(signature = (start, stop, step=None))] fn gen_series(start: PyExpr, stop: PyExpr, step: Option) -> PyExpr { let mut args = vec![start.into(), stop.into()]; if let Some(step) = step { args.push(step.into()); } Expr::ScalarFunction(datafusion::logical_expr::expr::ScalarFunction::new_udf( datafusion::functions_nested::range::gen_series_udf(), args, )) .into() } #[pyfunction] fn make_map(keys: Vec, values: Vec) -> PyExpr { let keys = keys.into_iter().map(|x| x.into()).collect(); let values = values.into_iter().map(|x| x.into()).collect(); datafusion::functions_nested::map::map(keys, values).into() } #[pyfunction] #[pyo3(signature = (array, element, index=None))] fn array_position(array: PyExpr, element: PyExpr, index: Option) -> PyExpr { let index = ScalarValue::Int64(index); let index = Expr::Literal(index, None); datafusion::functions_nested::expr_fn::array_position(array.into(), element.into(), index) .into() } #[pyfunction] #[pyo3(signature = (array, begin, end, stride=None))] fn array_slice(array: PyExpr, begin: PyExpr, end: PyExpr, stride: Option) -> PyExpr { datafusion::functions_nested::expr_fn::array_slice( array.into(), begin.into(), end.into(), stride.map(Into::into), ) .into() } /// Create a lambda expression from a list of parameter names and a body /// expression. The body should reference the parameters via [`lambda_var`]. /// Exposed to Python as `lambda_` because `lambda` is a reserved keyword. #[pyfunction] #[pyo3(name = "lambda_")] fn py_lambda(params: Vec, body: PyExpr) -> PyExpr { datafusion::logical_expr::lambda(params, body.into()).into() } /// Create an unresolved lambda variable reference by name. The owning /// higher-order function resolves it against its lambda parameters during /// planning. #[pyfunction] fn lambda_var(name: String) -> PyExpr { datafusion::logical_expr::lambda_var(name).into() } /// Higher-order function: apply `transform` (a lambda) to each element of /// `array`, returning a new array of the results. #[pyfunction] fn array_transform(array: PyExpr, transform: PyExpr) -> PyExpr { datafusion::functions_nested::expr_fn::array_transform(array.into(), transform.into()).into() } /// Higher-order function: return true if any element of `array` satisfies /// `predicate` (a lambda returning a boolean). #[pyfunction] fn array_any_match(array: PyExpr, predicate: PyExpr) -> PyExpr { datafusion::functions_nested::expr_fn::array_any_match(array.into(), predicate.into()).into() } /// Higher-order function: keep the elements of `array` for which `predicate` /// (a lambda returning a boolean) is true, returning a new filtered array. #[pyfunction] fn array_filter(array: PyExpr, predicate: PyExpr) -> PyExpr { datafusion::functions_nested::expr_fn::array_filter(array.into(), predicate.into()).into() } /// Computes a binary hash of the given data. type is the algorithm to use. /// Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, and blake3. // #[pyfunction(value, method)] #[pyfunction] fn digest(value: PyExpr, method: PyExpr) -> PyExpr { PyExpr { expr: functions::expr_fn::digest(value.expr, method.expr), } } /// Concatenates the text representations of all the arguments. /// NULL arguments are ignored. #[pyfunction] fn concat(args: Vec) -> PyResult { let args = args.into_iter().map(|e| e.expr).collect::>(); Ok(functions::string::expr_fn::concat(args).into()) } /// Concatenates all but the first argument, with separators. /// The first argument is used as the separator string, and should not be NULL. /// Other NULL arguments are ignored. #[pyfunction] fn concat_ws(sep: String, args: Vec) -> PyResult { let args = args.into_iter().map(|e| e.expr).collect::>(); Ok(functions::string::expr_fn::concat_ws(lit(sep), args).into()) } #[pyfunction] #[pyo3(signature = (values, regex, flags=None))] fn regexp_like(values: PyExpr, regex: PyExpr, flags: Option) -> PyResult { Ok(functions::expr_fn::regexp_like(values.expr, regex.expr, flags.map(|x| x.expr)).into()) } #[pyfunction] #[pyo3(signature = (values, regex, flags=None))] fn regexp_match(values: PyExpr, regex: PyExpr, flags: Option) -> PyResult { Ok(functions::expr_fn::regexp_match(values.expr, regex.expr, flags.map(|x| x.expr)).into()) } #[pyfunction] #[pyo3(signature = (string, pattern, replacement, flags=None))] /// Replaces substring(s) matching a POSIX regular expression. fn regexp_replace( string: PyExpr, pattern: PyExpr, replacement: PyExpr, flags: Option, ) -> PyResult { Ok(functions::expr_fn::regexp_replace( string.into(), pattern.into(), replacement.into(), flags.map(|x| x.expr), ) .into()) } #[pyfunction] #[pyo3(signature = (string, pattern, start, flags=None))] /// Returns the number of matches found in the string. fn regexp_count( string: PyExpr, pattern: PyExpr, start: Option, flags: Option, ) -> PyResult { Ok(functions::expr_fn::regexp_count( string.expr, pattern.expr, start.map(|x| x.expr), flags.map(|x| x.expr), ) .into()) } #[pyfunction] #[pyo3(signature = (values, regex, start=None, n=None, flags=None, subexpr=None))] /// Returns the position in a string where the specified occurrence of a regular expression is located fn regexp_instr( values: PyExpr, regex: PyExpr, start: Option, n: Option, flags: Option, subexpr: Option, ) -> PyResult { Ok(functions::expr_fn::regexp_instr( values.into(), regex.into(), start.map(|x| x.expr).or(Some(lit(1))), n.map(|x| x.expr).or(Some(lit(1))), None, flags.map(|x| x.expr).or(Some(lit(""))), subexpr.map(|x| x.expr).or(Some(lit(0))), ) .into()) } /// Creates a new Sort Expr #[pyfunction] fn order_by(expr: PyExpr, asc: bool, nulls_first: bool) -> PyResult { Ok(PySortExpr::from(datafusion::logical_expr::expr::Sort { expr: expr.expr, asc, nulls_first, })) } /// Creates a new Alias Expr #[pyfunction] #[pyo3(signature = (expr, name, metadata=None))] fn alias(expr: PyExpr, name: &str, metadata: Option>) -> PyResult { let relation: Option = None; let metadata = metadata.map(|m| FieldMetadata::new(m.into_iter().collect())); Ok(PyExpr { expr: datafusion::logical_expr::Expr::Alias( Alias::new(expr.expr, relation, name).with_metadata(metadata), ), }) } /// Create a column reference Expr #[pyfunction] fn col(name: &str) -> PyResult { Ok(PyExpr { expr: datafusion::logical_expr::Expr::Column(Column::new_unqualified(name)), }) } /// Create a CASE WHEN statement with literal WHEN expressions for comparison to the base expression. #[pyfunction] fn case(expr: PyExpr) -> PyResult { Ok(PyCaseBuilder::new(Some(expr))) } /// Create a CASE WHEN statement with literal WHEN expressions for comparison to the base expression. #[pyfunction] fn when(when: PyExpr, then: PyExpr) -> PyResult { Ok(PyCaseBuilder::new(None).when(when, then)) } // Generates a [pyo3] wrapper for associated aggregate functions. // All of the builder options are exposed to the python internal // function and we rely on the wrappers to only use those that // are appropriate. macro_rules! aggregate_function { ($NAME: ident) => { aggregate_function!($NAME, expr); }; ($NAME: ident, $($arg:ident)*) => { #[pyfunction] #[pyo3(signature = ($($arg),*, distinct=None, filter=None, order_by=None, null_treatment=None))] fn $NAME( $($arg: PyExpr),*, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option ) -> PyDataFusionResult { let agg_fn = functions_aggregate::expr_fn::$NAME($($arg.into()),*); add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) } }; } /// Generates a [pyo3] wrapper for [datafusion::functions::expr_fn] /// /// These functions have explicit named arguments. macro_rules! expr_fn { ($FUNC: ident) => { expr_fn!($FUNC, , stringify!($FUNC)); }; ($FUNC:ident, $($arg:ident)*) => { expr_fn!($FUNC, $($arg)*, stringify!($FUNC)); }; ($FUNC: ident, $DOC: expr) => { expr_fn!($FUNC, ,$DOC); }; ($FUNC: ident, $($arg:ident)*, $DOC: expr) => { #[doc = $DOC] #[pyfunction] fn $FUNC($($arg: PyExpr),*) -> PyExpr { functions::expr_fn::$FUNC($($arg.into()),*).into() } }; } /// Generates a [pyo3] wrapper for [datafusion::functions::expr_fn] /// /// These functions take a single `Vec` argument using `pyo3(signature = (*args))`. macro_rules! expr_fn_vec { ($FUNC: ident) => { expr_fn_vec!($FUNC, stringify!($FUNC)); }; ($FUNC: ident, $DOC: expr) => { #[doc = $DOC] #[pyfunction] #[pyo3(signature = (*args))] fn $FUNC(args: Vec) -> PyExpr { let args = args.into_iter().map(|e| e.into()).collect::>(); functions::expr_fn::$FUNC(args).into() } }; } /// Generates a [pyo3] wrapper for [datafusion_functions_nested::expr_fn] /// /// These functions have explicit named arguments. macro_rules! array_fn { ($FUNC: ident) => { array_fn!($FUNC, , stringify!($FUNC)); }; ($FUNC:ident, $($arg:ident)*) => { array_fn!($FUNC, $($arg)*, stringify!($FUNC)); }; ($FUNC: ident, $DOC: expr) => { array_fn!($FUNC, , $DOC); }; ($FUNC: ident, $($arg:ident)*, $DOC:expr) => { #[doc = $DOC] #[pyfunction] fn $FUNC($($arg: PyExpr),*) -> PyExpr { datafusion::functions_nested::expr_fn::$FUNC($($arg.into()),*).into() } }; } expr_fn!(abs, num); expr_fn!(acos, num); expr_fn!(acosh, num); expr_fn!( ascii, arg1, "Returns the numeric code of the first character of the argument. In UTF8 encoding, returns the Unicode code point of the character. In other multibyte encodings, the argument must be an ASCII character." ); expr_fn!(asin, num); expr_fn!(asinh, num); expr_fn!(atan, num); expr_fn!(atanh, num); expr_fn!(atan2, y x); expr_fn!( bit_length, arg, "Returns number of bits in the string (8 times the octet_length)." ); expr_fn_vec!( btrim, "Removes the longest string containing only characters in characters (a space by default) from the start and end of string." ); expr_fn!(cbrt, num); expr_fn!(ceil, num); expr_fn!( character_length, string, "Returns number of characters in the string." ); expr_fn!(length, string); expr_fn!(char_length, string); expr_fn!(chr, arg, "Returns the character with the given code."); expr_fn_vec!(coalesce); expr_fn_vec!(greatest); expr_fn_vec!(least); expr_fn!( contains, string search_str, "Return true if search_str is found within string (case-sensitive)." ); expr_fn!(cos, num); expr_fn!(cosh, num); expr_fn!(cot, num); expr_fn!(degrees, num); expr_fn!(decode, input encoding); expr_fn!(encode, input encoding); expr_fn!(ends_with, string suffix, "Returns true if string ends with suffix."); expr_fn!(exp, num); expr_fn!(factorial, num); expr_fn!(floor, num); expr_fn!(gcd, x y); expr_fn!( initcap, string, "Converts the first letter of each word to upper case and the rest to lower case. Words are sequences of alphanumeric characters separated by non-alphanumeric characters." ); expr_fn!(isnan, num); expr_fn!(iszero, num); expr_fn!(levenshtein, string1 string2); expr_fn!(lcm, x y); expr_fn!(left, string n, "Returns first n characters in the string, or when n is negative, returns all but last |n| characters."); expr_fn!(ln, num); expr_fn!(log, base num); expr_fn!(log10, num); expr_fn!(log2, num); expr_fn!(lower, arg1, "Converts the string to all lower case"); expr_fn_vec!( lpad, "Extends the string to length length by prepending the characters fill (a space by default). If the string is already longer than length then it is truncated (on the right)." ); expr_fn_vec!( ltrim, "Removes the longest string containing only characters in characters (a space by default) from the start of string." ); expr_fn!( md5, input_arg, "Computes the MD5 hash of the argument, with the result written in hexadecimal." ); expr_fn!( nanvl, x y, "Returns x if x is not NaN otherwise returns y." ); expr_fn!( nvl, x y, "Returns x if x is not NULL otherwise returns y." ); expr_fn!( nvl2, x y z, "Returns y if x is not NULL; otherwise returns z." ); expr_fn!(nullif, arg_1 arg_2); expr_fn!( octet_length, args, "Returns number of bytes in the string. Since this version of the function accepts type character directly, it will not strip trailing spaces." ); expr_fn_vec!(overlay); expr_fn!(pi); expr_fn!(power, base exponent); expr_fn!(radians, num); expr_fn!(repeat, string n, "Repeats string the specified number of times."); expr_fn!( replace, string from to, "Replaces all occurrences in string of substring from with substring to." ); expr_fn!( reverse, string, "Reverses the order of the characters in the string." ); expr_fn!(right, string n, "Returns last n characters in the string, or when n is negative, returns all but first |n| characters."); expr_fn_vec!(round); expr_fn_vec!( rpad, "Extends the string to length length by appending the characters fill (a space by default). If the string is already longer than length then it is truncated." ); expr_fn_vec!( rtrim, "Removes the longest string containing only characters in characters (a space by default) from the end of string." ); expr_fn!(sha224, input_arg1); expr_fn!(sha256, input_arg1); expr_fn!(sha384, input_arg1); expr_fn!(sha512, input_arg1); expr_fn!(signum, num); expr_fn!(sin, num); expr_fn!(sinh, num); expr_fn!( split_part, string delimiter index, "Splits string at occurrences of delimiter and returns the n'th field (counting from one)." ); expr_fn!(sqrt, num); expr_fn!(starts_with, string prefix, "Returns true if string starts with prefix."); expr_fn!(strpos, string substring, "Returns starting index of specified substring within string, or zero if it's not present. (Same as position(substring in string), but note the reversed argument order.)"); expr_fn!(substr, string position); expr_fn!(substr_index, string delimiter count); expr_fn!(substring, string position length); expr_fn!(find_in_set, string string_list); expr_fn!(tan, num); expr_fn!(tanh, num); expr_fn!( to_hex, arg1, "Converts the number to its equivalent hexadecimal representation." ); expr_fn!(now); expr_fn_vec!(to_date); expr_fn_vec!(to_local_time); expr_fn_vec!(to_time); expr_fn_vec!(to_timestamp); expr_fn_vec!(to_timestamp_millis); expr_fn_vec!(to_timestamp_nanos); expr_fn_vec!(to_timestamp_micros); expr_fn_vec!(to_timestamp_seconds); expr_fn_vec!(to_unixtime); expr_fn!(current_date); expr_fn!(current_time); expr_fn!(date_part, part date); expr_fn!(date_trunc, part date); expr_fn!(date_bin, stride source origin); expr_fn!(make_date, year month day); expr_fn!(make_time, hour minute second); expr_fn!(to_char, datetime format); expr_fn!(translate, string from to, "Replaces each character in string that matches a character in the from set with the corresponding character in the to set. If from is longer than to, occurrences of the extra characters in from are deleted."); expr_fn_vec!( trim, "Removes the longest string containing only characters in characters (a space by default) from the start, end, or both ends (BOTH is the default) of string." ); expr_fn_vec!(trunc); expr_fn!(upper, arg1, "Converts the string to all upper case."); expr_fn!(uuid); expr_fn_vec!(r#struct); // Use raw identifier since struct is a keyword expr_fn_vec!(named_struct); expr_fn!(from_unixtime, unixtime); expr_fn!(arrow_typeof, arg_1); expr_fn!(arrow_cast, arg_1 datatype); expr_fn!(arrow_try_cast, arg_1 datatype); expr_fn!(arrow_field, arg_1); expr_fn!(cast_to_type, arg_1 reference); expr_fn!(try_cast_to_type, arg_1 reference); expr_fn_vec!(arrow_metadata); expr_fn_vec!(with_metadata); expr_fn!(union_tag, arg1); expr_fn!(random); #[pyfunction] fn get_field(expr: PyExpr, names: Vec) -> PyExpr { let mut args = vec![expr.into()]; args.extend(names.into_iter().map(Into::into)); functions::core::get_field().call(args).into() } #[pyfunction] fn union_extract(union_expr: PyExpr, field_name: PyExpr) -> PyExpr { functions::core::union_extract() .call(vec![union_expr.into(), field_name.into()]) .into() } #[pyfunction] fn version() -> PyExpr { functions::core::version().call(vec![]).into() } // Array Functions array_fn!(array_append, array element); array_fn!(array_to_string, array delimiter); array_fn!(array_dims, array); array_fn!(array_distinct, array); array_fn!(array_element, array element); array_fn!(array_empty, array); array_fn!(array_length, array); array_fn!(array_has, first_array second_array); array_fn!(array_has_all, first_array second_array); array_fn!(array_has_any, first_array second_array); array_fn!(array_positions, array element); array_fn!(array_ndims, array); array_fn!(array_prepend, element array); array_fn!(array_pop_back, array); array_fn!(array_pop_front, array); array_fn!(array_remove, array element); array_fn!(array_remove_n, array element max); array_fn!(array_remove_all, array element); array_fn!(array_repeat, element count); array_fn!(array_replace, array from to); array_fn!(array_replace_n, array from to max); array_fn!(array_replace_all, array from to); array_fn!(array_sort, array desc null_first); array_fn!(array_compact, array); array_fn!(array_normalize, array); array_fn!(cosine_distance, array1 array2); array_fn!(inner_product, array1 array2); array_fn!(array_intersect, first_array second_array); array_fn!(array_union, array1 array2); array_fn!(array_except, first_array second_array); array_fn!(array_resize, array size value); array_fn!(array_any_value, array); array_fn!(array_max, array); array_fn!(array_min, array); array_fn!(array_reverse, array); array_fn!(cardinality, array); array_fn!(flatten, array); array_fn!(range, start stop step); // Map Functions array_fn!(map_keys, map); array_fn!(map_values, map); array_fn!(map_extract, map key); array_fn!(map_entries, map); aggregate_function!(array_agg); aggregate_function!(max); aggregate_function!(min); aggregate_function!(avg); aggregate_function!(sum); aggregate_function!(bit_and); aggregate_function!(bit_or); aggregate_function!(bit_xor); aggregate_function!(bool_and); aggregate_function!(bool_or); aggregate_function!(corr, y x); aggregate_function!(count); aggregate_function!(covar_samp, y x); aggregate_function!(covar_pop, y x); aggregate_function!(median); aggregate_function!(regr_slope, y x); aggregate_function!(regr_intercept, y x); aggregate_function!(regr_count, y x); aggregate_function!(regr_r2, y x); aggregate_function!(regr_avgx, y x); aggregate_function!(regr_avgy, y x); aggregate_function!(regr_sxx, y x); aggregate_function!(regr_syy, y x); aggregate_function!(regr_sxy, y x); aggregate_function!(stddev); aggregate_function!(stddev_pop); aggregate_function!(var_sample); aggregate_function!(var_pop); aggregate_function!(approx_distinct); aggregate_function!(approx_median); // The grouping function's physical plan is not implemented, but the // ResolveGroupingFunction analyzer rule rewrites it before the physical // planner sees it, so it works correctly at runtime. aggregate_function!(grouping); #[pyfunction] #[pyo3(signature = (sort_expression, percentile, num_centroids=None, filter=None))] pub fn approx_percentile_cont( sort_expression: PySortExpr, percentile: f64, num_centroids: Option, // enforces optional arguments at the end, currently filter: Option, ) -> PyDataFusionResult { let agg_fn = functions_aggregate::expr_fn::approx_percentile_cont( sort_expression.sort, lit(percentile), num_centroids.map(lit), ); add_builder_fns_to_aggregate(agg_fn, None, filter, None, None) } #[pyfunction] #[pyo3(signature = (sort_expression, weight, percentile, num_centroids=None, filter=None))] pub fn approx_percentile_cont_with_weight( sort_expression: PySortExpr, weight: PyExpr, percentile: f64, num_centroids: Option, filter: Option, ) -> PyDataFusionResult { let agg_fn = functions_aggregate::expr_fn::approx_percentile_cont_with_weight( sort_expression.sort, weight.expr, lit(percentile), num_centroids.map(lit), ); add_builder_fns_to_aggregate(agg_fn, None, filter, None, None) } #[pyfunction] #[pyo3(signature = (sort_expression, percentile, filter=None))] pub fn percentile_cont( sort_expression: PySortExpr, percentile: f64, filter: Option, ) -> PyDataFusionResult { let agg_fn = functions_aggregate::expr_fn::percentile_cont(sort_expression.sort, lit(percentile)); add_builder_fns_to_aggregate(agg_fn, None, filter, None, None) } // We handle last_value explicitly because the signature expects an order_by // https://github.com/apache/datafusion/issues/12376 #[pyfunction] #[pyo3(signature = (expr, distinct=None, filter=None, order_by=None, null_treatment=None))] pub fn last_value( expr: PyExpr, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { // If we initialize the UDAF with order_by directly, then it gets over-written by the builder let agg_fn = functions_aggregate::expr_fn::last_value(expr.expr, vec![]); add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) } // We handle first_value explicitly because the signature expects an order_by // https://github.com/apache/datafusion/issues/12376 #[pyfunction] #[pyo3(signature = (expr, distinct=None, filter=None, order_by=None, null_treatment=None))] pub fn first_value( expr: PyExpr, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { // If we initialize the UDAF with order_by directly, then it gets over-written by the builder let agg_fn = functions_aggregate::expr_fn::first_value(expr.expr, vec![]); add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) } // nth_value requires a non-expr argument #[pyfunction] #[pyo3(signature = (expr, n, distinct=None, filter=None, order_by=None, null_treatment=None))] pub fn nth_value( expr: PyExpr, n: i64, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { let agg_fn = datafusion::functions_aggregate::nth_value::nth_value(expr.expr, n, vec![]); add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) } // string_agg requires a non-expr argument #[pyfunction] #[pyo3(signature = (expr, delimiter, distinct=None, filter=None, order_by=None, null_treatment=None))] pub fn string_agg( expr: PyExpr, delimiter: String, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { let agg_fn = datafusion::functions_aggregate::string_agg::string_agg(expr.expr, lit(delimiter)); add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) } pub(crate) fn add_builder_fns_to_window( window_fn: Expr, partition_by: Option>, window_frame: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { let null_treatment = null_treatment.map(|n| n.into()); let mut builder = window_fn.null_treatment(null_treatment); if let Some(partition_cols) = partition_by { builder = builder.partition_by( partition_cols .into_iter() .map(|col| col.clone().into()) .collect(), ); } if let Some(order_by_cols) = order_by { let order_by_cols = to_sort_expressions(order_by_cols); builder = builder.order_by(order_by_cols); } if let Some(window_frame) = window_frame { builder = builder.window_frame(window_frame.into()); } Ok(builder.build().map(|e| e.into())?) } #[pyfunction] #[pyo3(signature = (arg, shift_offset, default_value=None, partition_by=None, order_by=None))] pub fn lead( arg: PyExpr, shift_offset: i64, default_value: Option, partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let default_value = default_value.map(|v| v.into()); let window_fn = functions_window::expr_fn::lead(arg.expr, Some(shift_offset), default_value); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (arg, shift_offset, default_value=None, partition_by=None, order_by=None))] pub fn lag( arg: PyExpr, shift_offset: i64, default_value: Option, partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let default_value = default_value.map(|v| v.into()); let window_fn = functions_window::expr_fn::lag(arg.expr, Some(shift_offset), default_value); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (partition_by=None, order_by=None))] pub fn row_number( partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let window_fn = functions_window::expr_fn::row_number(); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (partition_by=None, order_by=None))] pub fn rank( partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let window_fn = functions_window::expr_fn::rank(); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (partition_by=None, order_by=None))] pub fn dense_rank( partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let window_fn = functions_window::expr_fn::dense_rank(); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (partition_by=None, order_by=None))] pub fn percent_rank( partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let window_fn = functions_window::expr_fn::percent_rank(); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (partition_by=None, order_by=None))] pub fn cume_dist( partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let window_fn = functions_window::expr_fn::cume_dist(); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } #[pyfunction] #[pyo3(signature = (arg, partition_by=None, order_by=None))] pub fn ntile( arg: PyExpr, partition_by: Option>, order_by: Option>, ) -> PyDataFusionResult { let window_fn = functions_window::expr_fn::ntile(arg.into()); add_builder_fns_to_window(window_fn, partition_by, None, order_by, None) } pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_wrapped(wrap_pyfunction!(abs))?; m.add_wrapped(wrap_pyfunction!(acos))?; m.add_wrapped(wrap_pyfunction!(acosh))?; m.add_wrapped(wrap_pyfunction!(approx_distinct))?; m.add_wrapped(wrap_pyfunction!(alias))?; m.add_wrapped(wrap_pyfunction!(approx_median))?; m.add_wrapped(wrap_pyfunction!(approx_percentile_cont))?; m.add_wrapped(wrap_pyfunction!(approx_percentile_cont_with_weight))?; m.add_wrapped(wrap_pyfunction!(percentile_cont))?; m.add_wrapped(wrap_pyfunction!(range))?; m.add_wrapped(wrap_pyfunction!(array_agg))?; m.add_wrapped(wrap_pyfunction!(arrow_typeof))?; m.add_wrapped(wrap_pyfunction!(arrow_cast))?; m.add_wrapped(wrap_pyfunction!(arrow_try_cast))?; m.add_wrapped(wrap_pyfunction!(arrow_field))?; m.add_wrapped(wrap_pyfunction!(cast_to_type))?; m.add_wrapped(wrap_pyfunction!(try_cast_to_type))?; m.add_wrapped(wrap_pyfunction!(arrow_metadata))?; m.add_wrapped(wrap_pyfunction!(with_metadata))?; m.add_wrapped(wrap_pyfunction!(ascii))?; m.add_wrapped(wrap_pyfunction!(asin))?; m.add_wrapped(wrap_pyfunction!(asinh))?; m.add_wrapped(wrap_pyfunction!(atan))?; m.add_wrapped(wrap_pyfunction!(atanh))?; m.add_wrapped(wrap_pyfunction!(atan2))?; m.add_wrapped(wrap_pyfunction!(avg))?; m.add_wrapped(wrap_pyfunction!(bit_length))?; m.add_wrapped(wrap_pyfunction!(btrim))?; m.add_wrapped(wrap_pyfunction!(cbrt))?; m.add_wrapped(wrap_pyfunction!(ceil))?; m.add_wrapped(wrap_pyfunction!(character_length))?; m.add_wrapped(wrap_pyfunction!(chr))?; m.add_wrapped(wrap_pyfunction!(char_length))?; m.add_wrapped(wrap_pyfunction!(coalesce))?; m.add_wrapped(wrap_pyfunction!(case))?; m.add_wrapped(wrap_pyfunction!(when))?; m.add_wrapped(wrap_pyfunction!(col))?; m.add_wrapped(wrap_pyfunction!(concat_ws))?; m.add_wrapped(wrap_pyfunction!(concat))?; m.add_wrapped(wrap_pyfunction!(contains))?; m.add_wrapped(wrap_pyfunction!(corr))?; m.add_wrapped(wrap_pyfunction!(cos))?; m.add_wrapped(wrap_pyfunction!(cosh))?; m.add_wrapped(wrap_pyfunction!(cot))?; m.add_wrapped(wrap_pyfunction!(count))?; m.add_wrapped(wrap_pyfunction!(covar_pop))?; m.add_wrapped(wrap_pyfunction!(covar_samp))?; m.add_wrapped(wrap_pyfunction!(current_date))?; m.add_wrapped(wrap_pyfunction!(current_time))?; m.add_wrapped(wrap_pyfunction!(degrees))?; m.add_wrapped(wrap_pyfunction!(date_bin))?; m.add_wrapped(wrap_pyfunction!(date_part))?; m.add_wrapped(wrap_pyfunction!(date_trunc))?; m.add_wrapped(wrap_pyfunction!(make_date))?; m.add_wrapped(wrap_pyfunction!(make_time))?; m.add_wrapped(wrap_pyfunction!(digest))?; m.add_wrapped(wrap_pyfunction!(ends_with))?; m.add_wrapped(wrap_pyfunction!(exp))?; m.add_wrapped(wrap_pyfunction!(factorial))?; m.add_wrapped(wrap_pyfunction!(floor))?; m.add_wrapped(wrap_pyfunction!(from_unixtime))?; m.add_wrapped(wrap_pyfunction!(gcd))?; m.add_wrapped(wrap_pyfunction!(greatest))?; m.add_wrapped(wrap_pyfunction!(grouping))?; m.add_wrapped(wrap_pyfunction!(in_list))?; m.add_wrapped(wrap_pyfunction!(initcap))?; m.add_wrapped(wrap_pyfunction!(isnan))?; m.add_wrapped(wrap_pyfunction!(iszero))?; m.add_wrapped(wrap_pyfunction!(levenshtein))?; m.add_wrapped(wrap_pyfunction!(lcm))?; m.add_wrapped(wrap_pyfunction!(least))?; m.add_wrapped(wrap_pyfunction!(left))?; m.add_wrapped(wrap_pyfunction!(length))?; m.add_wrapped(wrap_pyfunction!(ln))?; m.add_wrapped(wrap_pyfunction!(self::log))?; m.add_wrapped(wrap_pyfunction!(log10))?; m.add_wrapped(wrap_pyfunction!(log2))?; m.add_wrapped(wrap_pyfunction!(lower))?; m.add_wrapped(wrap_pyfunction!(lpad))?; m.add_wrapped(wrap_pyfunction!(ltrim))?; m.add_wrapped(wrap_pyfunction!(max))?; m.add_wrapped(wrap_pyfunction!(make_array))?; m.add_wrapped(wrap_pyfunction!(md5))?; m.add_wrapped(wrap_pyfunction!(median))?; m.add_wrapped(wrap_pyfunction!(min))?; m.add_wrapped(wrap_pyfunction!(named_struct))?; m.add_wrapped(wrap_pyfunction!(nanvl))?; m.add_wrapped(wrap_pyfunction!(nvl))?; m.add_wrapped(wrap_pyfunction!(nvl2))?; m.add_wrapped(wrap_pyfunction!(now))?; m.add_wrapped(wrap_pyfunction!(nullif))?; m.add_wrapped(wrap_pyfunction!(octet_length))?; m.add_wrapped(wrap_pyfunction!(order_by))?; m.add_wrapped(wrap_pyfunction!(overlay))?; m.add_wrapped(wrap_pyfunction!(pi))?; m.add_wrapped(wrap_pyfunction!(power))?; m.add_wrapped(wrap_pyfunction!(radians))?; m.add_wrapped(wrap_pyfunction!(random))?; m.add_wrapped(wrap_pyfunction!(regexp_count))?; m.add_wrapped(wrap_pyfunction!(regexp_instr))?; m.add_wrapped(wrap_pyfunction!(regexp_like))?; m.add_wrapped(wrap_pyfunction!(regexp_match))?; m.add_wrapped(wrap_pyfunction!(regexp_replace))?; m.add_wrapped(wrap_pyfunction!(repeat))?; m.add_wrapped(wrap_pyfunction!(replace))?; m.add_wrapped(wrap_pyfunction!(reverse))?; m.add_wrapped(wrap_pyfunction!(right))?; m.add_wrapped(wrap_pyfunction!(round))?; m.add_wrapped(wrap_pyfunction!(rpad))?; m.add_wrapped(wrap_pyfunction!(rtrim))?; m.add_wrapped(wrap_pyfunction!(sha224))?; m.add_wrapped(wrap_pyfunction!(sha256))?; m.add_wrapped(wrap_pyfunction!(sha384))?; m.add_wrapped(wrap_pyfunction!(sha512))?; m.add_wrapped(wrap_pyfunction!(signum))?; m.add_wrapped(wrap_pyfunction!(sin))?; m.add_wrapped(wrap_pyfunction!(sinh))?; m.add_wrapped(wrap_pyfunction!(split_part))?; m.add_wrapped(wrap_pyfunction!(sqrt))?; m.add_wrapped(wrap_pyfunction!(starts_with))?; m.add_wrapped(wrap_pyfunction!(stddev))?; m.add_wrapped(wrap_pyfunction!(stddev_pop))?; m.add_wrapped(wrap_pyfunction!(string_agg))?; m.add_wrapped(wrap_pyfunction!(strpos))?; m.add_wrapped(wrap_pyfunction!(r#struct))?; // Use raw identifier since struct is a keyword m.add_wrapped(wrap_pyfunction!(substr))?; m.add_wrapped(wrap_pyfunction!(substr_index))?; m.add_wrapped(wrap_pyfunction!(substring))?; m.add_wrapped(wrap_pyfunction!(find_in_set))?; m.add_wrapped(wrap_pyfunction!(sum))?; m.add_wrapped(wrap_pyfunction!(tan))?; m.add_wrapped(wrap_pyfunction!(tanh))?; m.add_wrapped(wrap_pyfunction!(to_hex))?; m.add_wrapped(wrap_pyfunction!(to_char))?; m.add_wrapped(wrap_pyfunction!(to_date))?; m.add_wrapped(wrap_pyfunction!(to_local_time))?; m.add_wrapped(wrap_pyfunction!(to_time))?; m.add_wrapped(wrap_pyfunction!(to_timestamp))?; m.add_wrapped(wrap_pyfunction!(to_timestamp_millis))?; m.add_wrapped(wrap_pyfunction!(to_timestamp_nanos))?; m.add_wrapped(wrap_pyfunction!(to_timestamp_micros))?; m.add_wrapped(wrap_pyfunction!(to_timestamp_seconds))?; m.add_wrapped(wrap_pyfunction!(to_unixtime))?; m.add_wrapped(wrap_pyfunction!(translate))?; m.add_wrapped(wrap_pyfunction!(trim))?; m.add_wrapped(wrap_pyfunction!(trunc))?; m.add_wrapped(wrap_pyfunction!(upper))?; m.add_wrapped(wrap_pyfunction!(get_field))?; m.add_wrapped(wrap_pyfunction!(union_extract))?; m.add_wrapped(wrap_pyfunction!(union_tag))?; m.add_wrapped(wrap_pyfunction!(version))?; m.add_wrapped(wrap_pyfunction!(self::uuid))?; // Use self to avoid name collision m.add_wrapped(wrap_pyfunction!(var_pop))?; m.add_wrapped(wrap_pyfunction!(var_sample))?; m.add_wrapped(wrap_pyfunction!(regr_avgx))?; m.add_wrapped(wrap_pyfunction!(regr_avgy))?; m.add_wrapped(wrap_pyfunction!(regr_count))?; m.add_wrapped(wrap_pyfunction!(regr_intercept))?; m.add_wrapped(wrap_pyfunction!(regr_r2))?; m.add_wrapped(wrap_pyfunction!(regr_slope))?; m.add_wrapped(wrap_pyfunction!(regr_sxx))?; m.add_wrapped(wrap_pyfunction!(regr_sxy))?; m.add_wrapped(wrap_pyfunction!(regr_syy))?; m.add_wrapped(wrap_pyfunction!(first_value))?; m.add_wrapped(wrap_pyfunction!(last_value))?; m.add_wrapped(wrap_pyfunction!(nth_value))?; m.add_wrapped(wrap_pyfunction!(bit_and))?; m.add_wrapped(wrap_pyfunction!(bit_or))?; m.add_wrapped(wrap_pyfunction!(bit_xor))?; m.add_wrapped(wrap_pyfunction!(bool_and))?; m.add_wrapped(wrap_pyfunction!(bool_or))?; //Binary String Functions m.add_wrapped(wrap_pyfunction!(encode))?; m.add_wrapped(wrap_pyfunction!(decode))?; // Lambda / higher-order functions m.add_wrapped(wrap_pyfunction!(py_lambda))?; m.add_wrapped(wrap_pyfunction!(lambda_var))?; m.add_wrapped(wrap_pyfunction!(array_transform))?; m.add_wrapped(wrap_pyfunction!(array_any_match))?; m.add_wrapped(wrap_pyfunction!(array_filter))?; // Array Functions m.add_wrapped(wrap_pyfunction!(array_append))?; m.add_wrapped(wrap_pyfunction!(array_concat))?; m.add_wrapped(wrap_pyfunction!(array_cat))?; m.add_wrapped(wrap_pyfunction!(array_dims))?; m.add_wrapped(wrap_pyfunction!(array_distinct))?; m.add_wrapped(wrap_pyfunction!(array_compact))?; m.add_wrapped(wrap_pyfunction!(array_normalize))?; m.add_wrapped(wrap_pyfunction!(cosine_distance))?; m.add_wrapped(wrap_pyfunction!(inner_product))?; m.add_wrapped(wrap_pyfunction!(array_element))?; m.add_wrapped(wrap_pyfunction!(array_empty))?; m.add_wrapped(wrap_pyfunction!(array_length))?; m.add_wrapped(wrap_pyfunction!(array_has))?; m.add_wrapped(wrap_pyfunction!(array_has_all))?; m.add_wrapped(wrap_pyfunction!(array_has_any))?; m.add_wrapped(wrap_pyfunction!(array_position))?; m.add_wrapped(wrap_pyfunction!(array_positions))?; m.add_wrapped(wrap_pyfunction!(array_to_string))?; m.add_wrapped(wrap_pyfunction!(array_intersect))?; m.add_wrapped(wrap_pyfunction!(array_union))?; m.add_wrapped(wrap_pyfunction!(array_except))?; m.add_wrapped(wrap_pyfunction!(array_resize))?; m.add_wrapped(wrap_pyfunction!(array_ndims))?; m.add_wrapped(wrap_pyfunction!(array_prepend))?; m.add_wrapped(wrap_pyfunction!(array_pop_back))?; m.add_wrapped(wrap_pyfunction!(array_pop_front))?; m.add_wrapped(wrap_pyfunction!(array_remove))?; m.add_wrapped(wrap_pyfunction!(array_remove_n))?; m.add_wrapped(wrap_pyfunction!(array_remove_all))?; m.add_wrapped(wrap_pyfunction!(array_repeat))?; m.add_wrapped(wrap_pyfunction!(array_replace))?; m.add_wrapped(wrap_pyfunction!(array_replace_n))?; m.add_wrapped(wrap_pyfunction!(array_replace_all))?; m.add_wrapped(wrap_pyfunction!(array_sort))?; m.add_wrapped(wrap_pyfunction!(array_slice))?; m.add_wrapped(wrap_pyfunction!(array_any_value))?; m.add_wrapped(wrap_pyfunction!(array_distance))?; m.add_wrapped(wrap_pyfunction!(array_max))?; m.add_wrapped(wrap_pyfunction!(array_min))?; m.add_wrapped(wrap_pyfunction!(array_reverse))?; m.add_wrapped(wrap_pyfunction!(arrays_zip))?; m.add_wrapped(wrap_pyfunction!(string_to_array))?; m.add_wrapped(wrap_pyfunction!(gen_series))?; m.add_wrapped(wrap_pyfunction!(flatten))?; m.add_wrapped(wrap_pyfunction!(cardinality))?; // Map Functions m.add_wrapped(wrap_pyfunction!(make_map))?; m.add_wrapped(wrap_pyfunction!(map_keys))?; m.add_wrapped(wrap_pyfunction!(map_values))?; m.add_wrapped(wrap_pyfunction!(map_extract))?; m.add_wrapped(wrap_pyfunction!(map_entries))?; // Window Functions m.add_wrapped(wrap_pyfunction!(lead))?; m.add_wrapped(wrap_pyfunction!(lag))?; m.add_wrapped(wrap_pyfunction!(rank))?; m.add_wrapped(wrap_pyfunction!(row_number))?; m.add_wrapped(wrap_pyfunction!(dense_rank))?; m.add_wrapped(wrap_pyfunction!(percent_rank))?; m.add_wrapped(wrap_pyfunction!(cume_dist))?; m.add_wrapped(wrap_pyfunction!(ntile))?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/lib.rs000066400000000000000000000114341521727353300226220ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. // Re-export Apache Arrow DataFusion dependencies pub use datafusion::{ self, common as datafusion_common, logical_expr as datafusion_expr, optimizer, sql as datafusion_sql, }; #[cfg(feature = "substrait")] pub use datafusion_substrait; #[cfg(feature = "mimalloc")] use mimalloc::MiMalloc; use pyo3::prelude::*; pub mod analyzer; pub mod catalog; pub mod codec; pub mod common; pub mod context; pub mod dataframe; mod dataset; mod dataset_exec; pub mod errors; pub mod expr; mod functions; pub mod metrics; mod options; pub mod physical_plan; mod pyarrow_filter_expression; pub mod pyarrow_util; mod record_batch; mod spark_functions; pub mod sql; pub mod store; pub mod table; pub mod unparser; mod array; #[cfg(feature = "substrait")] pub mod substrait; mod udaf; mod udf; pub mod udtf; mod udwf; // Re-export helpers previously consumed by downstream Rust crates. // Modules stay private to keep the public Rust API surface small. pub use udaf::to_rust_accumulator; pub use udwf::{MultiColumnWindowUDF, PythonFunctionWindowUDF, to_rust_partition_evaluator}; #[cfg(feature = "mimalloc")] #[global_allocator] static GLOBAL: MiMalloc = MiMalloc; /// Low-level DataFusion internal package. /// /// The higher-level public API is defined in pure python files under the /// datafusion directory. #[pymodule] fn _internal(py: Python, m: Bound<'_, PyModule>) -> PyResult<()> { // Initialize logging pyo3_log::init(); // Register the python classes m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; let catalog = PyModule::new(py, "catalog")?; catalog::init_module(&catalog)?; m.add_submodule(&catalog)?; // Register `common` as a submodule. Matching `datafusion-common` https://docs.rs/datafusion-common/latest/datafusion_common/ let common = PyModule::new(py, "common")?; common::init_module(&common)?; m.add_submodule(&common)?; // Register `expr` as a submodule. Matching `datafusion-expr` https://docs.rs/datafusion-expr/latest/datafusion_expr/ let expr = PyModule::new(py, "expr")?; expr::init_module(&expr)?; m.add_submodule(&expr)?; let unparser = PyModule::new(py, "unparser")?; unparser::init_module(&unparser)?; m.add_submodule(&unparser)?; // Register the functions as a submodule let funcs = PyModule::new(py, "functions")?; functions::init_module(&funcs)?; // Spark-compatible functions live under `functions.spark`. let spark_funcs = PyModule::new(py, "spark")?; spark_functions::init_module(&spark_funcs)?; funcs.add_submodule(&spark_funcs)?; m.add_submodule(&funcs)?; let store = PyModule::new(py, "object_store")?; store::init_module(&store)?; m.add_submodule(&store)?; let options = PyModule::new(py, "options")?; options::init_module(&options)?; m.add_submodule(&options)?; // Register substrait as a submodule #[cfg(feature = "substrait")] setup_substrait_module(py, &m)?; Ok(()) } #[cfg(feature = "substrait")] fn setup_substrait_module(py: Python, m: &Bound<'_, PyModule>) -> PyResult<()> { let substrait = PyModule::new(py, "substrait")?; substrait::init_module(&substrait)?; m.add_submodule(&substrait)?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/metrics.rs000066400000000000000000000124151521727353300235220ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use std::sync::Arc; use chrono::{Datelike, Timelike}; use datafusion::physical_plan::metrics::{Metric, MetricValue, MetricsSet, Timestamp}; use pyo3::prelude::*; #[pyclass(from_py_object, frozen, name = "MetricsSet", module = "datafusion")] #[derive(Debug, Clone)] pub struct PyMetricsSet { metrics: MetricsSet, } impl PyMetricsSet { pub fn new(metrics: MetricsSet) -> Self { Self { metrics } } } #[pymethods] impl PyMetricsSet { fn metrics(&self) -> Vec { self.metrics .iter() .map(|m| PyMetric::new(Arc::clone(m))) .collect() } fn output_rows(&self) -> Option { self.metrics.output_rows() } fn elapsed_compute(&self) -> Option { self.metrics.elapsed_compute() } fn spill_count(&self) -> Option { self.metrics.spill_count() } fn spilled_bytes(&self) -> Option { self.metrics.spilled_bytes() } fn spilled_rows(&self) -> Option { self.metrics.spilled_rows() } fn sum_by_name(&self, name: &str) -> Option { self.metrics.sum_by_name(name).map(|v| v.as_usize()) } fn __repr__(&self) -> String { format!("{}", self.metrics) } } #[pyclass(from_py_object, frozen, name = "Metric", module = "datafusion")] #[derive(Debug, Clone)] pub struct PyMetric { metric: Arc, } impl PyMetric { pub fn new(metric: Arc) -> Self { Self { metric } } fn timestamp_to_pyobject<'py>( py: Python<'py>, ts: &Timestamp, ) -> PyResult>> { match ts.value() { Some(dt) => { let datetime_mod = py.import("datetime")?; let datetime_cls = datetime_mod.getattr("datetime")?; let tz_utc = datetime_mod.getattr("timezone")?.getattr("utc")?; let result = datetime_cls.call1(( dt.year(), dt.month(), dt.day(), dt.hour(), dt.minute(), dt.second(), dt.timestamp_subsec_micros(), tz_utc, ))?; Ok(Some(result)) } None => Ok(None), } } } #[pymethods] impl PyMetric { #[getter] fn name(&self) -> String { self.metric.value().name().to_string() } #[getter] fn value<'py>(&self, py: Python<'py>) -> PyResult>> { match self.metric.value() { MetricValue::OutputRows(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), MetricValue::OutputBytes(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), MetricValue::ElapsedCompute(t) => Ok(Some(t.value().into_pyobject(py)?.into_any())), MetricValue::SpillCount(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), MetricValue::SpilledBytes(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), MetricValue::SpilledRows(c) => Ok(Some(c.value().into_pyobject(py)?.into_any())), MetricValue::CurrentMemoryUsage(g) => Ok(Some(g.value().into_pyobject(py)?.into_any())), MetricValue::Count { count, .. } => { Ok(Some(count.value().into_pyobject(py)?.into_any())) } MetricValue::Gauge { gauge, .. } => { Ok(Some(gauge.value().into_pyobject(py)?.into_any())) } MetricValue::Time { time, .. } => Ok(Some(time.value().into_pyobject(py)?.into_any())), MetricValue::StartTimestamp(ts) | MetricValue::EndTimestamp(ts) => { Self::timestamp_to_pyobject(py, ts) } _ => Ok(None), } } #[getter] fn value_as_datetime<'py>(&self, py: Python<'py>) -> PyResult>> { match self.metric.value() { MetricValue::StartTimestamp(ts) | MetricValue::EndTimestamp(ts) => { Self::timestamp_to_pyobject(py, ts) } _ => Ok(None), } } #[getter] fn partition(&self) -> Option { self.metric.partition() } fn labels(&self) -> HashMap { self.metric .labels() .iter() .map(|l| (l.name().to_string(), l.value().to_string())) .collect() } fn __repr__(&self) -> String { format!("{}", self.metric.value()) } } apache-datafusion-python-54.0.0/crates/core/src/options.rs000066400000000000000000000123171521727353300235500ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use arrow::datatypes::{DataType, Schema}; use arrow::pyarrow::PyArrowType; use datafusion::prelude::CsvReadOptions; use pyo3::prelude::{PyModule, PyModuleMethods}; use pyo3::{Bound, PyResult, pyclass, pymethods}; use crate::context::parse_file_compression_type; use crate::errors::PyDataFusionError; use crate::expr::sort_expr::PySortExpr; /// Options for reading CSV files #[pyclass(name = "CsvReadOptions", module = "datafusion.options", frozen)] pub struct PyCsvReadOptions { pub has_header: bool, pub delimiter: u8, pub quote: u8, pub terminator: Option, pub escape: Option, pub comment: Option, pub newlines_in_values: bool, pub schema: Option>, pub schema_infer_max_records: usize, pub file_extension: String, pub table_partition_cols: Vec<(String, PyArrowType)>, pub file_compression_type: String, pub file_sort_order: Vec>, pub null_regex: Option, pub truncated_rows: bool, } #[pymethods] impl PyCsvReadOptions { #[allow(clippy::too_many_arguments)] #[pyo3(signature = ( has_header=true, delimiter=b',', quote=b'"', terminator=None, escape=None, comment=None, newlines_in_values=false, schema=None, schema_infer_max_records=1000, file_extension=".csv".to_string(), table_partition_cols=vec![], file_compression_type="".to_string(), file_sort_order=vec![], null_regex=None, truncated_rows=false ))] #[new] fn new( has_header: bool, delimiter: u8, quote: u8, terminator: Option, escape: Option, comment: Option, newlines_in_values: bool, schema: Option>, schema_infer_max_records: usize, file_extension: String, table_partition_cols: Vec<(String, PyArrowType)>, file_compression_type: String, file_sort_order: Vec>, null_regex: Option, truncated_rows: bool, ) -> Self { Self { has_header, delimiter, quote, terminator, escape, comment, newlines_in_values, schema, schema_infer_max_records, file_extension, table_partition_cols, file_compression_type, file_sort_order, null_regex, truncated_rows, } } } impl<'a> TryFrom<&'a PyCsvReadOptions> for CsvReadOptions<'a> { type Error = PyDataFusionError; fn try_from(value: &'a PyCsvReadOptions) -> Result, Self::Error> { let partition_cols: Vec<(String, DataType)> = value .table_partition_cols .iter() .map(|(name, dtype)| (name.clone(), dtype.0.clone())) .collect(); let compression = parse_file_compression_type(Some(value.file_compression_type.clone()))?; let sort_order: Vec> = value .file_sort_order .iter() .map(|inner| { inner .iter() .map(|sort_expr| sort_expr.sort.clone()) .collect() }) .collect(); // Explicit struct initialization to catch upstream changes let mut options = CsvReadOptions { has_header: value.has_header, delimiter: value.delimiter, quote: value.quote, terminator: value.terminator, escape: value.escape, comment: value.comment, newlines_in_values: value.newlines_in_values, schema: None, // Will be set separately due to lifetime constraints schema_infer_max_records: value.schema_infer_max_records, file_extension: value.file_extension.as_str(), table_partition_cols: partition_cols, file_compression_type: compression, file_sort_order: sort_order, null_regex: value.null_regex.clone(), truncated_rows: value.truncated_rows, }; // Set schema separately to handle the lifetime options.schema = value.schema.as_ref().map(|s| &s.0); Ok(options) } } pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/physical_plan.rs000066400000000000000000000104221521727353300246760ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::physical_plan::{ExecutionPlan, ExecutionPlanProperties, displayable}; use datafusion_proto::physical_plan::AsExecutionPlan; use prost::Message; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use pyo3::types::PyBytes; use crate::codec::PythonPhysicalCodec; use crate::context::PySessionContext; use crate::errors::PyDataFusionResult; use crate::metrics::PyMetricsSet; #[pyclass( from_py_object, frozen, name = "ExecutionPlan", module = "datafusion", subclass )] #[derive(Debug, Clone)] pub struct PyExecutionPlan { pub plan: Arc, } impl PyExecutionPlan { /// creates a new PyPhysicalPlan pub fn new(plan: Arc) -> Self { Self { plan } } } #[pymethods] impl PyExecutionPlan { /// Get the inputs to this plan pub fn children(&self) -> Vec { self.plan .children() .iter() .map(|&p| p.to_owned().into()) .collect() } pub fn display(&self) -> String { let d = displayable(self.plan.as_ref()); format!("{}", d.one_line()) } pub fn display_indent(&self) -> String { let d = displayable(self.plan.as_ref()); format!("{}", d.indent(false)) } #[pyo3(signature = (ctx=None))] pub fn to_bytes<'py>( &'py self, py: Python<'py>, ctx: Option, ) -> PyDataFusionResult> { // Route through the session's physical codec when supplied so // user FFI codecs registered via // `with_physical_extension_codec` see the encode path. let default_codec; let codec: &dyn datafusion_proto::physical_plan::PhysicalExtensionCodec = match ctx { Some(ref ctx) => ctx.physical_codec().as_ref(), None => { default_codec = PythonPhysicalCodec::default(); &default_codec } }; let proto = datafusion_proto::protobuf::PhysicalPlanNode::try_from_physical_plan( self.plan.clone(), codec, )?; let bytes = proto.encode_to_vec(); Ok(PyBytes::new(py, &bytes)) } #[staticmethod] pub fn from_bytes( ctx: PySessionContext, proto_msg: Bound<'_, PyBytes>, ) -> PyDataFusionResult { let bytes: &[u8] = proto_msg.extract().map_err(Into::::into)?; let proto_plan = datafusion_proto::protobuf::PhysicalPlanNode::decode(bytes).map_err(|e| { PyRuntimeError::new_err(format!( "Unable to decode physical node from serialized bytes: {e}" )) })?; let codec = ctx.physical_codec(); let plan = proto_plan.try_into_physical_plan(ctx.ctx.task_ctx().as_ref(), codec.as_ref())?; Ok(Self::new(plan)) } pub fn metrics(&self) -> Option { self.plan.metrics().map(PyMetricsSet::new) } fn __repr__(&self) -> String { self.display_indent() } #[getter] pub fn partition_count(&self) -> usize { self.plan.output_partitioning().partition_count() } } impl From for Arc { fn from(plan: PyExecutionPlan) -> Arc { plan.plan.clone() } } impl From> for PyExecutionPlan { fn from(plan: Arc) -> PyExecutionPlan { PyExecutionPlan { plan: plan.clone() } } } apache-datafusion-python-54.0.0/crates/core/src/pyarrow_filter_expression.rs000066400000000000000000000175041521727353300274070ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::convert::TryFrom; use std::result::Result; use datafusion::common::{Column, ScalarValue}; use datafusion::logical_expr::expr::InList; use datafusion::logical_expr::{Between, BinaryExpr, Expr, Operator}; /// Converts a Datafusion logical plan expression (Expr) into a PyArrow compute expression use pyo3::{IntoPyObjectExt, prelude::*}; use crate::errors::{PyDataFusionError, PyDataFusionResult}; use crate::pyarrow_util::scalar_to_pyarrow; #[derive(Debug)] #[repr(transparent)] pub(crate) struct PyArrowFilterExpression(Py); fn operator_to_py<'py>( operator: &Operator, op: &Bound<'py, PyModule>, ) -> PyDataFusionResult> { let py_op: Bound<'_, PyAny> = match operator { Operator::Eq => op.getattr("eq")?, Operator::NotEq => op.getattr("ne")?, Operator::Lt => op.getattr("lt")?, Operator::LtEq => op.getattr("le")?, Operator::Gt => op.getattr("gt")?, Operator::GtEq => op.getattr("ge")?, Operator::And => op.getattr("and_")?, Operator::Or => op.getattr("or_")?, _ => { return Err(PyDataFusionError::Common(format!( "Unsupported operator {operator:?}" ))); } }; Ok(py_op) } fn extract_scalar_list<'py>( exprs: &[Expr], py: Python<'py>, ) -> PyDataFusionResult>> { exprs .iter() .map(|expr| match expr { // TODO: should we also leverage `ScalarValue::to_pyarrow` here? Expr::Literal(v, _) => match v { // The unwraps here are for infallible conversions ScalarValue::Boolean(Some(b)) => Ok(b.into_bound_py_any(py)?), ScalarValue::Int8(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::Int16(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::Int32(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::Int64(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::UInt8(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::UInt16(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::UInt32(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::UInt64(Some(i)) => Ok(i.into_bound_py_any(py)?), ScalarValue::Float32(Some(f)) => Ok(f.into_bound_py_any(py)?), ScalarValue::Float64(Some(f)) => Ok(f.into_bound_py_any(py)?), ScalarValue::Utf8(Some(s)) => Ok(s.into_bound_py_any(py)?), _ => Err(PyDataFusionError::Common(format!( "PyArrow can't handle ScalarValue: {v:?}" ))), }, _ => Err(PyDataFusionError::Common(format!( "Only a list of Literals are supported got {expr:?}" ))), }) .collect() } impl PyArrowFilterExpression { pub fn inner(&self) -> &Py { &self.0 } } impl TryFrom<&Expr> for PyArrowFilterExpression { type Error = PyDataFusionError; // Converts a Datafusion filter Expr into an expression string that can be evaluated by Python // Note that pyarrow.compute.{field,scalar} are put into Python globals() when evaluated // isin, is_null, and is_valid (~is_null) are methods of pyarrow.dataset.Expression // https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Expression.html#pyarrow-dataset-expression fn try_from(expr: &Expr) -> Result { Python::attach(|py| { let pc = Python::import(py, "pyarrow.compute")?; let op_module = Python::import(py, "operator")?; let pc_expr: PyDataFusionResult> = match expr { Expr::Column(Column { name, .. }) => Ok(pc.getattr("field")?.call1((name,))?), Expr::Literal(scalar, _) => Ok(scalar_to_pyarrow(scalar, py)?), Expr::BinaryExpr(BinaryExpr { left, op, right }) => { let operator = operator_to_py(op, &op_module)?; let left = PyArrowFilterExpression::try_from(left.as_ref())?.0; let right = PyArrowFilterExpression::try_from(right.as_ref())?.0; Ok(operator.call1((left, right))?) } Expr::Not(expr) => { let operator = op_module.getattr("invert")?; let py_expr = PyArrowFilterExpression::try_from(expr.as_ref())?.0; Ok(operator.call1((py_expr,))?) } Expr::IsNotNull(expr) => { let py_expr = PyArrowFilterExpression::try_from(expr.as_ref())? .0 .into_bound(py); Ok(py_expr.call_method0("is_valid")?) } Expr::IsNull(expr) => { let expr = PyArrowFilterExpression::try_from(expr.as_ref())? .0 .into_bound(py); // https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Expression.html#pyarrow.dataset.Expression.is_null // Whether floating-point NaNs are considered null. let nan_is_null = false; let res = expr.call_method1("is_null", (nan_is_null,))?; Ok(res) } Expr::Between(Between { expr, negated, low, high, }) => { let expr = PyArrowFilterExpression::try_from(expr.as_ref())?.0; let low = PyArrowFilterExpression::try_from(low.as_ref())?.0; let high = PyArrowFilterExpression::try_from(high.as_ref())?.0; let and = op_module.getattr("and_")?; let le = op_module.getattr("le")?; let invert = op_module.getattr("invert")?; // scalar <= field() returns a boolean expression so we need to use and to combine these let ret = and.call1(( le.call1((low, expr.clone_ref(py)))?, le.call1((expr, high))?, ))?; Ok(if *negated { invert.call1((ret,))? } else { ret }) } Expr::InList(InList { expr, list, negated, }) => { let expr = PyArrowFilterExpression::try_from(expr.as_ref())? .0 .into_bound(py); let scalars = extract_scalar_list(list, py)?; let ret = expr.call_method1("isin", (scalars,))?; let invert = op_module.getattr("invert")?; Ok(if *negated { invert.call1((ret,))? } else { ret }) } _ => Err(PyDataFusionError::Common(format!( "Unsupported Datafusion expression {expr:?}" ))), }; Ok(PyArrowFilterExpression(pc_expr?.into())) }) } } apache-datafusion-python-54.0.0/crates/core/src/pyarrow_util.rs000066400000000000000000000145261521727353300246210ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! Conversions between PyArrow and DataFusion types use std::sync::Arc; use arrow::array::{Array, ArrayData, ArrayRef, ListArray, make_array}; use arrow::buffer::OffsetBuffer; use arrow::datatypes::Field; use arrow::pyarrow::{FromPyArrow, ToPyArrow}; use datafusion::common::exec_err; use datafusion::scalar::ScalarValue; use pyo3::types::{PyAnyMethods, PyList}; use pyo3::{Borrowed, Bound, FromPyObject, PyAny, PyErr, PyResult, Python}; use crate::common::data_type::PyScalarValue; use crate::errors::PyDataFusionError; /// Helper function to turn an Array into a ScalarValue. If ``as_list_array`` is true, /// the array will be turned into a ``ListArray``. Otherwise, we extract the first value /// from the array. fn array_to_scalar_value(array: ArrayRef, as_list_array: bool) -> PyResult { if as_list_array { let field = Arc::new(Field::new_list_field( array.data_type().clone(), array.nulls().is_some(), )); let offsets = OffsetBuffer::from_lengths(vec![array.len()]); let list_array = ListArray::new(field, offsets, array, None); Ok(PyScalarValue(ScalarValue::List(Arc::new(list_array)))) } else { let scalar = ScalarValue::try_from_array(&array, 0).map_err(PyDataFusionError::from)?; Ok(PyScalarValue(scalar)) } } /// Helper function to take any Python object that contains an Arrow PyCapsule /// interface and attempt to extract a scalar value from it. If `as_list_array` /// is true, the array will be turned into a ``ListArray``. Otherwise, we extract /// the first value from the array. fn pyobj_extract_scalar_via_capsule( value: &Bound<'_, PyAny>, as_list_array: bool, ) -> PyResult { let array_data = ArrayData::from_pyarrow_bound(value)?; let array = make_array(array_data); array_to_scalar_value(array, as_list_array) } impl FromPyArrow for PyScalarValue { fn from_pyarrow_bound(value: &Bound<'_, PyAny>) -> PyResult { let py = value.py(); let pyarrow_mod = py.import("pyarrow"); // Is it a PyArrow object? if let Ok(pa) = pyarrow_mod.as_ref() { let scalar_type = pa.getattr("Scalar")?; if value.is_instance(&scalar_type)? { let typ = value.getattr("type")?; // construct pyarrow array from the python value and pyarrow type let factory = py.import("pyarrow")?.getattr("array")?; let args = PyList::new(py, [value])?; let array = factory.call1((args, typ))?; return pyobj_extract_scalar_via_capsule(&array, false); } let array_type = pa.getattr("Array")?; if value.is_instance(&array_type)? { return pyobj_extract_scalar_via_capsule(value, true); } } // Is it a NanoArrow scalar? if let Ok(na) = py.import("nanoarrow") { let scalar_type = py.import("nanoarrow.array")?.getattr("Scalar")?; if value.is_instance(&scalar_type)? { return pyobj_extract_scalar_via_capsule(value, false); } let array_type = na.getattr("Array")?; if value.is_instance(&array_type)? { return pyobj_extract_scalar_via_capsule(value, true); } } // Is it a arro3 scalar? if let Ok(arro3) = py.import("arro3").and_then(|arro3| arro3.getattr("core")) { let scalar_type = arro3.getattr("Scalar")?; if value.is_instance(&scalar_type)? { return pyobj_extract_scalar_via_capsule(value, false); } let array_type = arro3.getattr("Array")?; if value.is_instance(&array_type)? { return pyobj_extract_scalar_via_capsule(value, true); } } // Does it have a PyCapsule interface but isn't one of our known libraries? // If so do our "best guess". Try checking type name, and if that fails // return a single value if the length is 1 and return a List value otherwise if value.hasattr("__arrow_c_array__")? { let type_name = value.get_type().repr()?; if type_name.contains("Scalar")? { return pyobj_extract_scalar_via_capsule(value, false); } if type_name.contains("Array")? { return pyobj_extract_scalar_via_capsule(value, true); } let array_data = ArrayData::from_pyarrow_bound(value)?; let array = make_array(array_data); let as_array_list = array.len() != 1; return array_to_scalar_value(array, as_array_list); } // Last attempt - try to create a PyArrow scalar from a plain Python object if let Ok(pa) = pyarrow_mod.as_ref() { let scalar = pa.call_method1("scalar", (value,))?; PyScalarValue::from_pyarrow_bound(&scalar) } else { exec_err!("Unable to import scalar value").map_err(PyDataFusionError::from)? } } } impl<'source> FromPyObject<'_, 'source> for PyScalarValue { type Error = PyErr; fn extract(value: Borrowed<'_, 'source, PyAny>) -> Result { Self::from_pyarrow_bound(&value) } } pub fn scalar_to_pyarrow<'py>( scalar: &ScalarValue, py: Python<'py>, ) -> PyResult> { let array = scalar.to_array().map_err(PyDataFusionError::from)?; // convert to pyarrow array using C data interface let pyarray = array.to_data().to_pyarrow(py)?; let pyscalar = pyarray.call_method1("__getitem__", (0,))?; Ok(pyscalar) } apache-datafusion-python-54.0.0/crates/core/src/record_batch.rs000066400000000000000000000070361521727353300244760ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::arrow::pyarrow::ToPyArrow; use datafusion::arrow::record_batch::RecordBatch; use datafusion::physical_plan::SendableRecordBatchStream; use datafusion_python_util::wait_for_future; use futures::StreamExt; use pyo3::exceptions::{PyStopAsyncIteration, PyStopIteration}; use pyo3::prelude::*; use pyo3::{PyAny, PyResult, Python, pyclass, pymethods}; use tokio::sync::Mutex; use crate::errors::PyDataFusionError; #[pyclass(name = "RecordBatch", module = "datafusion", subclass, frozen)] pub struct PyRecordBatch { batch: RecordBatch, } #[pymethods] impl PyRecordBatch { fn to_pyarrow<'py>(&self, py: Python<'py>) -> PyResult> { self.batch.to_pyarrow(py) } } impl From for PyRecordBatch { fn from(batch: RecordBatch) -> Self { Self { batch } } } #[pyclass(name = "RecordBatchStream", module = "datafusion", subclass, frozen)] pub struct PyRecordBatchStream { stream: Arc>, } impl PyRecordBatchStream { pub fn new(stream: SendableRecordBatchStream) -> Self { Self { stream: Arc::new(Mutex::new(stream)), } } } #[pymethods] impl PyRecordBatchStream { fn next(&self, py: Python) -> PyResult { let stream = self.stream.clone(); wait_for_future(py, next_stream(stream, true))? } fn __next__(&self, py: Python) -> PyResult { self.next(py) } fn __anext__<'py>(&'py self, py: Python<'py>) -> PyResult> { let stream = self.stream.clone(); pyo3_async_runtimes::tokio::future_into_py(py, next_stream(stream, false)) } fn __iter__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> { slf } fn __aiter__(slf: PyRef<'_, Self>) -> PyRef<'_, Self> { slf } } /// Polls the next batch from a `SendableRecordBatchStream`, converting the `Option>` form. pub(crate) async fn poll_next_batch( stream: &mut SendableRecordBatchStream, ) -> datafusion::error::Result> { stream.next().await.transpose() } async fn next_stream( stream: Arc>, sync: bool, ) -> PyResult { let mut stream = stream.lock().await; match poll_next_batch(&mut stream).await { Ok(Some(batch)) => Ok(batch.into()), Ok(None) => { // Depending on whether the iteration is sync or not, we raise either a // StopIteration or a StopAsyncIteration if sync { Err(PyStopIteration::new_err("stream exhausted")) } else { Err(PyStopAsyncIteration::new_err("stream exhausted")) } } Err(e) => Err(PyDataFusionError::from(e))?, } } apache-datafusion-python-54.0.0/crates/core/src/spark_functions.rs000066400000000000000000000340241521727353300252640ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. //! PyO3 wrappers for the [`datafusion-spark`] crate. //! //! Exposes Spark-compatible scalar and aggregate function builders for use //! from Python under `datafusion.functions.spark`. use datafusion::logical_expr::Expr; use datafusion::logical_expr::expr::ScalarFunction; use datafusion_spark::{expr_fn, function as udf}; use pyo3::prelude::*; use pyo3::wrap_pyfunction; use crate::common::data_type::NullTreatment; use crate::errors::PyDataFusionResult; use crate::expr::PyExpr; use crate::expr::sort_expr::PySortExpr; use crate::functions::add_builder_fns_to_aggregate; /// Generates a [pyo3] wrapper for [datafusion_spark::expr_fn]. /// /// These functions have explicit named arguments and mirror the upstream /// `expr_fn::$FUNC` signature. macro_rules! spark_expr_fn { ($FUNC:ident) => { spark_expr_fn!($FUNC,); }; ($FUNC:ident, $($arg:ident)*) => { #[pyfunction] fn $FUNC($($arg: PyExpr),*) -> PyExpr { expr_fn::$FUNC($($arg.into()),*).into() } }; } /// Generates a variadic [pyo3] wrapper that calls the [`ScalarUDF`] factory /// directly. Required for functions whose upstream `expr_fn` wrapper accepts /// a single `Expr` instead of `Vec` (an upstream `export_functions!` /// macro-arm quirk), so we bypass it to get true Python `*args` semantics. macro_rules! spark_udf_vec { ($PY_NAME:ident, $UDF_PATH:path) => { #[pyfunction] #[pyo3(signature = (*args))] fn $PY_NAME(args: Vec) -> PyExpr { let udf = $UDF_PATH(); let args: Vec = args.into_iter().map(Into::into).collect(); Expr::ScalarFunction(ScalarFunction::new_udf(udf, args)).into() } }; } /// Generates a [pyo3] wrapper for Spark aggregate functions. Mirrors /// [`crate::functions::aggregate_function`] but points at /// [`datafusion_spark::expr_fn`]. macro_rules! spark_aggregate { ($NAME:ident) => { spark_aggregate!($NAME, expr); }; ($NAME:ident, $($arg:ident)*) => { #[pyfunction] #[pyo3(signature = ($($arg),*, distinct=None, filter=None, order_by=None, null_treatment=None))] fn $NAME( $($arg: PyExpr),*, distinct: Option, filter: Option, order_by: Option>, null_treatment: Option, ) -> PyDataFusionResult { let agg_fn = expr_fn::$NAME($($arg.into()),*); add_builder_fns_to_aggregate(agg_fn, distinct, filter, order_by, null_treatment) } }; } // --------------------------------------------------------------------------- // Aggregate functions // --------------------------------------------------------------------------- spark_aggregate!(avg, arg1); spark_aggregate!(try_sum, arg1); spark_aggregate!(collect_list, arg1); spark_aggregate!(collect_set, arg1); // --------------------------------------------------------------------------- // Array functions // --------------------------------------------------------------------------- // Upstream factory is `spark_array_contains`; expose under the Spark SQL // name `array_contains` on the Python side. #[pyfunction] fn array_contains(arr: PyExpr, element: PyExpr) -> PyExpr { expr_fn::spark_array_contains(arr.into(), element.into()).into() } spark_udf_vec!(array, udf::array::array); spark_expr_fn!(shuffle, arg1); spark_expr_fn!(array_repeat, element count); spark_expr_fn!(slice, arr start length); // --------------------------------------------------------------------------- // Bitmap functions // --------------------------------------------------------------------------- spark_expr_fn!(bitmap_count, arg1); spark_expr_fn!(bitmap_bit_position, arg1); spark_expr_fn!(bitmap_bucket_number, arg1); // --------------------------------------------------------------------------- // Bitwise functions // --------------------------------------------------------------------------- spark_expr_fn!(bit_get, col pos); spark_expr_fn!(bit_count, col); spark_expr_fn!(bitwise_not, col); spark_expr_fn!(shiftleft, value shift); spark_expr_fn!(shiftright, value shift); spark_expr_fn!(shiftrightunsigned, value shift); // --------------------------------------------------------------------------- // Collection / Conditional / Conversion // --------------------------------------------------------------------------- spark_expr_fn!(size, arg1); // Python keyword `if` → exposed as `if_`. Upstream Rust ident is `r#if`. #[pyfunction] fn if_(condition: PyExpr, if_true: PyExpr, if_false: PyExpr) -> PyExpr { expr_fn::r#if(condition.into(), if_true.into(), if_false.into()).into() } // `spark_cast` is config-injected by the upstream `expr_fn` helper; defaults // applied automatically there. spark_expr_fn!(spark_cast, arg1 arg2); // --------------------------------------------------------------------------- // Datetime functions // --------------------------------------------------------------------------- spark_expr_fn!(add_months, start_date num_months); spark_expr_fn!(date_add, start_date days); spark_expr_fn!(date_sub, start_date days); spark_expr_fn!(hour, arg1); spark_expr_fn!(minute, arg1); spark_expr_fn!(second, arg1); spark_expr_fn!(last_day, arg1); spark_expr_fn!(make_dt_interval, days hours mins secs); spark_expr_fn!(make_interval, years months weeks days hours mins secs); spark_expr_fn!(next_day, start_date day_of_week); spark_expr_fn!(date_diff, end_date start_date); spark_expr_fn!(date_trunc, fmt ts); spark_expr_fn!(time_trunc, fmt t); spark_expr_fn!(trunc, dt fmt); spark_expr_fn!(date_part, field source); spark_expr_fn!(from_utc_timestamp, ts tz); spark_expr_fn!(to_utc_timestamp, ts tz); spark_expr_fn!(unix_date, dt); spark_expr_fn!(unix_micros, ts); spark_expr_fn!(unix_millis, ts); spark_expr_fn!(unix_seconds, ts); // --------------------------------------------------------------------------- // Hash functions // --------------------------------------------------------------------------- spark_expr_fn!(crc32, arg1); spark_expr_fn!(sha1, arg1); spark_expr_fn!(sha2, arg1 bit_length); spark_udf_vec!(xxhash64, udf::hash::xxhash64); // --------------------------------------------------------------------------- // JSON functions // --------------------------------------------------------------------------- spark_udf_vec!(json_tuple, udf::json::json_tuple); // --------------------------------------------------------------------------- // Map functions // --------------------------------------------------------------------------- spark_expr_fn!(map_from_arrays, keys values); spark_expr_fn!(map_from_entries, arg1); spark_expr_fn!(str_to_map, text pair_delim key_value_delim); // --------------------------------------------------------------------------- // Math functions // --------------------------------------------------------------------------- spark_expr_fn!(abs, arg1); spark_expr_fn!(ceil, arg1); spark_expr_fn!(expm1, arg1); spark_expr_fn!(factorial, arg1); spark_expr_fn!(floor, arg1); spark_expr_fn!(hex, arg1); spark_expr_fn!(modulus, dividend divisor); spark_expr_fn!(pmod, dividend divisor); spark_expr_fn!(rint, arg1); spark_expr_fn!(round, value scale); spark_expr_fn!(unhex, arg1); spark_expr_fn!(width_bucket, value min_value max_value num_buckets); spark_expr_fn!(csc, arg1); spark_expr_fn!(sec, arg1); spark_expr_fn!(negative, arg1); spark_expr_fn!(bin, arg1); // --------------------------------------------------------------------------- // String functions // --------------------------------------------------------------------------- spark_expr_fn!(ascii, arg1); spark_expr_fn!(base64, bin_input); // `char` collides with the Rust primitive type in macro hygiene; rename the // Rust ident and re-expose under the original name to Python. #[pyfunction] #[pyo3(name = "char")] fn char_fn(arg1: PyExpr) -> PyExpr { expr_fn::char(arg1.into()).into() } spark_udf_vec!(concat, udf::string::concat); spark_udf_vec!(elt, udf::string::elt); spark_expr_fn!(ilike, str pattern); spark_expr_fn!(length, arg1); spark_expr_fn!(like, str pattern); spark_expr_fn!(luhn_check, arg1); spark_udf_vec!(format_string, udf::string::format_string); spark_expr_fn!(space, arg1); spark_expr_fn!(substring, str pos length); spark_expr_fn!(unbase64, str); spark_expr_fn!(soundex, str); spark_expr_fn!(is_valid_utf8, str); spark_expr_fn!(make_valid_utf8, str); // --------------------------------------------------------------------------- // URL functions // --------------------------------------------------------------------------- spark_udf_vec!(parse_url, udf::url::parse_url); spark_udf_vec!(try_parse_url, udf::url::try_parse_url); spark_udf_vec!(url_decode, udf::url::url_decode); spark_udf_vec!(try_url_decode, udf::url::try_url_decode); spark_udf_vec!(url_encode, udf::url::url_encode); // --------------------------------------------------------------------------- // Module init // --------------------------------------------------------------------------- pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { // Aggregate m.add_wrapped(wrap_pyfunction!(avg))?; m.add_wrapped(wrap_pyfunction!(try_sum))?; m.add_wrapped(wrap_pyfunction!(collect_list))?; m.add_wrapped(wrap_pyfunction!(collect_set))?; // Array m.add_wrapped(wrap_pyfunction!(array_contains))?; m.add_wrapped(wrap_pyfunction!(array))?; m.add_wrapped(wrap_pyfunction!(shuffle))?; m.add_wrapped(wrap_pyfunction!(array_repeat))?; m.add_wrapped(wrap_pyfunction!(slice))?; // Bitmap m.add_wrapped(wrap_pyfunction!(bitmap_count))?; m.add_wrapped(wrap_pyfunction!(bitmap_bit_position))?; m.add_wrapped(wrap_pyfunction!(bitmap_bucket_number))?; // Bitwise m.add_wrapped(wrap_pyfunction!(bit_get))?; m.add_wrapped(wrap_pyfunction!(bit_count))?; m.add_wrapped(wrap_pyfunction!(bitwise_not))?; m.add_wrapped(wrap_pyfunction!(shiftleft))?; m.add_wrapped(wrap_pyfunction!(shiftright))?; m.add_wrapped(wrap_pyfunction!(shiftrightunsigned))?; // Collection m.add_wrapped(wrap_pyfunction!(size))?; // Conditional m.add_wrapped(wrap_pyfunction!(if_))?; // Conversion m.add_wrapped(wrap_pyfunction!(spark_cast))?; // Datetime m.add_wrapped(wrap_pyfunction!(add_months))?; m.add_wrapped(wrap_pyfunction!(date_add))?; m.add_wrapped(wrap_pyfunction!(date_sub))?; m.add_wrapped(wrap_pyfunction!(hour))?; m.add_wrapped(wrap_pyfunction!(minute))?; m.add_wrapped(wrap_pyfunction!(second))?; m.add_wrapped(wrap_pyfunction!(last_day))?; m.add_wrapped(wrap_pyfunction!(make_dt_interval))?; m.add_wrapped(wrap_pyfunction!(make_interval))?; m.add_wrapped(wrap_pyfunction!(next_day))?; m.add_wrapped(wrap_pyfunction!(date_diff))?; m.add_wrapped(wrap_pyfunction!(date_trunc))?; m.add_wrapped(wrap_pyfunction!(time_trunc))?; m.add_wrapped(wrap_pyfunction!(trunc))?; m.add_wrapped(wrap_pyfunction!(date_part))?; m.add_wrapped(wrap_pyfunction!(from_utc_timestamp))?; m.add_wrapped(wrap_pyfunction!(to_utc_timestamp))?; m.add_wrapped(wrap_pyfunction!(unix_date))?; m.add_wrapped(wrap_pyfunction!(unix_micros))?; m.add_wrapped(wrap_pyfunction!(unix_millis))?; m.add_wrapped(wrap_pyfunction!(unix_seconds))?; // Hash m.add_wrapped(wrap_pyfunction!(crc32))?; m.add_wrapped(wrap_pyfunction!(sha1))?; m.add_wrapped(wrap_pyfunction!(sha2))?; m.add_wrapped(wrap_pyfunction!(xxhash64))?; // JSON m.add_wrapped(wrap_pyfunction!(json_tuple))?; // Map m.add_wrapped(wrap_pyfunction!(map_from_arrays))?; m.add_wrapped(wrap_pyfunction!(map_from_entries))?; m.add_wrapped(wrap_pyfunction!(str_to_map))?; // Math m.add_wrapped(wrap_pyfunction!(abs))?; m.add_wrapped(wrap_pyfunction!(ceil))?; m.add_wrapped(wrap_pyfunction!(expm1))?; m.add_wrapped(wrap_pyfunction!(factorial))?; m.add_wrapped(wrap_pyfunction!(floor))?; m.add_wrapped(wrap_pyfunction!(hex))?; m.add_wrapped(wrap_pyfunction!(modulus))?; m.add_wrapped(wrap_pyfunction!(pmod))?; m.add_wrapped(wrap_pyfunction!(rint))?; m.add_wrapped(wrap_pyfunction!(round))?; m.add_wrapped(wrap_pyfunction!(unhex))?; m.add_wrapped(wrap_pyfunction!(width_bucket))?; m.add_wrapped(wrap_pyfunction!(csc))?; m.add_wrapped(wrap_pyfunction!(sec))?; m.add_wrapped(wrap_pyfunction!(negative))?; m.add_wrapped(wrap_pyfunction!(bin))?; // String m.add_wrapped(wrap_pyfunction!(ascii))?; m.add_wrapped(wrap_pyfunction!(base64))?; m.add_wrapped(wrap_pyfunction!(char_fn))?; m.add_wrapped(wrap_pyfunction!(concat))?; m.add_wrapped(wrap_pyfunction!(elt))?; m.add_wrapped(wrap_pyfunction!(ilike))?; m.add_wrapped(wrap_pyfunction!(length))?; m.add_wrapped(wrap_pyfunction!(like))?; m.add_wrapped(wrap_pyfunction!(luhn_check))?; m.add_wrapped(wrap_pyfunction!(format_string))?; m.add_wrapped(wrap_pyfunction!(space))?; m.add_wrapped(wrap_pyfunction!(substring))?; m.add_wrapped(wrap_pyfunction!(unbase64))?; m.add_wrapped(wrap_pyfunction!(soundex))?; m.add_wrapped(wrap_pyfunction!(is_valid_utf8))?; m.add_wrapped(wrap_pyfunction!(make_valid_utf8))?; // URL m.add_wrapped(wrap_pyfunction!(parse_url))?; m.add_wrapped(wrap_pyfunction!(try_parse_url))?; m.add_wrapped(wrap_pyfunction!(url_decode))?; m.add_wrapped(wrap_pyfunction!(try_url_decode))?; m.add_wrapped(wrap_pyfunction!(url_encode))?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/sql.rs000066400000000000000000000015341521727353300226530ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. pub mod exceptions; pub mod logical; pub(crate) mod util; apache-datafusion-python-54.0.0/crates/core/src/sql/000077500000000000000000000000001521727353300223025ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/src/sql/exceptions.rs000066400000000000000000000023231521727353300250310ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::fmt::{Debug, Display}; use pyo3::PyErr; pub fn py_type_err(e: impl Debug + Display) -> PyErr { PyErr::new::(format!("{e}")) } pub fn py_runtime_err(e: impl Debug + Display) -> PyErr { PyErr::new::(format!("{e}")) } pub fn py_value_err(e: impl Debug + Display) -> PyErr { PyErr::new::(format!("{e}")) } apache-datafusion-python-54.0.0/crates/core/src/sql/logical.rs000066400000000000000000000243521521727353300242700ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::logical_expr::{DdlStatement, LogicalPlan, Statement}; use datafusion_proto::logical_plan::AsLogicalPlan; use prost::Message; use pyo3::exceptions::PyRuntimeError; use pyo3::prelude::*; use pyo3::types::PyBytes; use crate::codec::PythonLogicalCodec; use crate::context::PySessionContext; use crate::errors::PyDataFusionResult; use crate::expr::aggregate::PyAggregate; use crate::expr::analyze::PyAnalyze; use crate::expr::copy_to::PyCopyTo; use crate::expr::create_catalog::PyCreateCatalog; use crate::expr::create_catalog_schema::PyCreateCatalogSchema; use crate::expr::create_external_table::PyCreateExternalTable; use crate::expr::create_function::PyCreateFunction; use crate::expr::create_index::PyCreateIndex; use crate::expr::create_memory_table::PyCreateMemoryTable; use crate::expr::create_view::PyCreateView; use crate::expr::describe_table::PyDescribeTable; use crate::expr::distinct::PyDistinct; use crate::expr::dml::PyDmlStatement; use crate::expr::drop_catalog_schema::PyDropCatalogSchema; use crate::expr::drop_function::PyDropFunction; use crate::expr::drop_table::PyDropTable; use crate::expr::drop_view::PyDropView; use crate::expr::empty_relation::PyEmptyRelation; use crate::expr::explain::PyExplain; use crate::expr::extension::PyExtension; use crate::expr::filter::PyFilter; use crate::expr::join::PyJoin; use crate::expr::limit::PyLimit; use crate::expr::logical_node::LogicalNode; use crate::expr::projection::PyProjection; use crate::expr::recursive_query::PyRecursiveQuery; use crate::expr::repartition::PyRepartition; use crate::expr::sort::PySort; use crate::expr::statement::{ PyDeallocate, PyExecute, PyPrepare, PyResetVariable, PySetVariable, PyTransactionEnd, PyTransactionStart, }; use crate::expr::subquery::PySubquery; use crate::expr::subquery_alias::PySubqueryAlias; use crate::expr::table_scan::PyTableScan; use crate::expr::union::PyUnion; use crate::expr::unnest::PyUnnest; use crate::expr::values::PyValues; use crate::expr::window::PyWindowExpr; #[pyclass( from_py_object, frozen, name = "LogicalPlan", module = "datafusion", subclass, eq )] #[derive(Debug, Clone, PartialEq, Eq)] pub struct PyLogicalPlan { pub(crate) plan: Arc, } impl PyLogicalPlan { /// creates a new PyLogicalPlan pub fn new(plan: LogicalPlan) -> Self { Self { plan: Arc::new(plan), } } pub fn plan(&self) -> Arc { self.plan.clone() } } #[pymethods] impl PyLogicalPlan { /// Return the specific logical operator pub fn to_variant<'py>(&self, py: Python<'py>) -> PyResult> { match self.plan.as_ref() { LogicalPlan::Aggregate(plan) => PyAggregate::from(plan.clone()).to_variant(py), LogicalPlan::Analyze(plan) => PyAnalyze::from(plan.clone()).to_variant(py), LogicalPlan::Distinct(plan) => PyDistinct::from(plan.clone()).to_variant(py), LogicalPlan::EmptyRelation(plan) => PyEmptyRelation::from(plan.clone()).to_variant(py), LogicalPlan::Explain(plan) => PyExplain::from(plan.clone()).to_variant(py), LogicalPlan::Extension(plan) => PyExtension::from(plan.clone()).to_variant(py), LogicalPlan::Filter(plan) => PyFilter::from(plan.clone()).to_variant(py), LogicalPlan::Join(plan) => PyJoin::from(plan.clone()).to_variant(py), LogicalPlan::Limit(plan) => PyLimit::from(plan.clone()).to_variant(py), LogicalPlan::Projection(plan) => PyProjection::from(plan.clone()).to_variant(py), LogicalPlan::Sort(plan) => PySort::from(plan.clone()).to_variant(py), LogicalPlan::TableScan(plan) => PyTableScan::from(plan.clone()).to_variant(py), LogicalPlan::Subquery(plan) => PySubquery::from(plan.clone()).to_variant(py), LogicalPlan::SubqueryAlias(plan) => PySubqueryAlias::from(plan.clone()).to_variant(py), LogicalPlan::Unnest(plan) => PyUnnest::from(plan.clone()).to_variant(py), LogicalPlan::Window(plan) => PyWindowExpr::from(plan.clone()).to_variant(py), LogicalPlan::Repartition(plan) => PyRepartition::from(plan.clone()).to_variant(py), LogicalPlan::Union(plan) => PyUnion::from(plan.clone()).to_variant(py), LogicalPlan::Statement(plan) => match plan { Statement::TransactionStart(plan) => { PyTransactionStart::from(plan.clone()).to_variant(py) } Statement::TransactionEnd(plan) => { PyTransactionEnd::from(plan.clone()).to_variant(py) } Statement::SetVariable(plan) => PySetVariable::from(plan.clone()).to_variant(py), Statement::ResetVariable(plan) => { PyResetVariable::from(plan.clone()).to_variant(py) } Statement::Prepare(plan) => PyPrepare::from(plan.clone()).to_variant(py), Statement::Execute(plan) => PyExecute::from(plan.clone()).to_variant(py), Statement::Deallocate(plan) => PyDeallocate::from(plan.clone()).to_variant(py), }, LogicalPlan::Values(plan) => PyValues::from(plan.clone()).to_variant(py), LogicalPlan::Dml(plan) => PyDmlStatement::from(plan.clone()).to_variant(py), LogicalPlan::Ddl(plan) => match plan { DdlStatement::CreateExternalTable(plan) => { PyCreateExternalTable::from(plan.clone()).to_variant(py) } DdlStatement::CreateMemoryTable(plan) => { PyCreateMemoryTable::from(plan.clone()).to_variant(py) } DdlStatement::CreateView(plan) => PyCreateView::from(plan.clone()).to_variant(py), DdlStatement::CreateCatalogSchema(plan) => { PyCreateCatalogSchema::from(plan.clone()).to_variant(py) } DdlStatement::CreateCatalog(plan) => { PyCreateCatalog::from(plan.clone()).to_variant(py) } DdlStatement::CreateIndex(plan) => PyCreateIndex::from(plan.clone()).to_variant(py), DdlStatement::DropTable(plan) => PyDropTable::from(plan.clone()).to_variant(py), DdlStatement::DropView(plan) => PyDropView::from(plan.clone()).to_variant(py), DdlStatement::DropCatalogSchema(plan) => { PyDropCatalogSchema::from(plan.clone()).to_variant(py) } DdlStatement::CreateFunction(plan) => { PyCreateFunction::from(plan.clone()).to_variant(py) } DdlStatement::DropFunction(plan) => { PyDropFunction::from(plan.clone()).to_variant(py) } }, LogicalPlan::Copy(plan) => PyCopyTo::from(plan.clone()).to_variant(py), LogicalPlan::DescribeTable(plan) => PyDescribeTable::from(plan.clone()).to_variant(py), LogicalPlan::RecursiveQuery(plan) => { PyRecursiveQuery::from(plan.clone()).to_variant(py) } } } /// Get the inputs to this plan fn inputs(&self) -> Vec { let mut inputs = vec![]; for input in self.plan.inputs() { inputs.push(input.to_owned().into()); } inputs } fn __repr__(&self) -> PyResult { Ok(format!("{:?}", self.plan)) } fn display(&self) -> String { format!("{}", self.plan.display()) } fn display_indent(&self) -> String { format!("{}", self.plan.display_indent()) } fn display_indent_schema(&self) -> String { format!("{}", self.plan.display_indent_schema()) } fn display_graphviz(&self) -> String { format!("{}", self.plan.display_graphviz()) } #[pyo3(signature = (ctx=None))] pub fn to_bytes<'py>( &'py self, py: Python<'py>, ctx: Option, ) -> PyDataFusionResult> { let default_codec; let codec: &dyn datafusion_proto::logical_plan::LogicalExtensionCodec = match ctx { Some(ref ctx) => ctx.logical_codec().as_ref(), None => { default_codec = PythonLogicalCodec::default(); &default_codec } }; let proto = datafusion_proto::protobuf::LogicalPlanNode::try_from_logical_plan(&self.plan, codec)?; let bytes = proto.encode_to_vec(); Ok(PyBytes::new(py, &bytes)) } #[staticmethod] pub fn from_bytes( ctx: PySessionContext, proto_msg: Bound<'_, PyBytes>, ) -> PyDataFusionResult { let bytes: &[u8] = proto_msg.extract().map_err(Into::::into)?; let proto_plan = datafusion_proto::protobuf::LogicalPlanNode::decode(bytes).map_err(|e| { PyRuntimeError::new_err(format!( "Unable to decode logical node from serialized bytes: {e}" )) })?; let codec = ctx.logical_codec(); let plan = proto_plan.try_into_logical_plan(&ctx.ctx.task_ctx(), codec.as_ref())?; Ok(Self::new(plan)) } } impl From for LogicalPlan { fn from(logical_plan: PyLogicalPlan) -> LogicalPlan { logical_plan.plan.as_ref().clone() } } impl From for PyLogicalPlan { fn from(logical_plan: LogicalPlan) -> PyLogicalPlan { PyLogicalPlan { plan: Arc::new(logical_plan), } } } apache-datafusion-python-54.0.0/crates/core/src/sql/util.rs000066400000000000000000000060111521727353300236230ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::collections::HashMap; use datafusion::common::{DataFusionError, exec_err, plan_datafusion_err}; use datafusion::logical_expr::sqlparser::dialect::dialect_from_str; use datafusion::sql::sqlparser::dialect::Dialect; use datafusion::sql::sqlparser::parser::Parser; use datafusion::sql::sqlparser::tokenizer::{Token, Tokenizer}; fn tokens_from_replacements( placeholder: &str, replacements: &HashMap>, ) -> Option> { if let Some(pattern) = placeholder.strip_prefix("$") { replacements.get(pattern).cloned() } else { None } } fn get_tokens_for_string_replacement( dialect: &dyn Dialect, replacements: HashMap, ) -> Result>, DataFusionError> { replacements .into_iter() .map(|(name, value)| { let tokens = Tokenizer::new(dialect, &value) .tokenize() .map_err(|err| DataFusionError::External(err.into()))?; Ok((name, tokens)) }) .collect() } pub(crate) fn replace_placeholders_with_strings( query: &str, dialect: &str, replacements: HashMap, ) -> Result { let dialect = dialect_from_str(dialect) .ok_or_else(|| plan_datafusion_err!("Unsupported SQL dialect: {dialect}."))?; let replacements = get_tokens_for_string_replacement(dialect.as_ref(), replacements)?; let tokens = Tokenizer::new(dialect.as_ref(), query) .tokenize() .map_err(|err| DataFusionError::External(err.into()))?; let replaced_tokens = tokens .into_iter() .flat_map(|token| { if let Token::Placeholder(placeholder) = &token { tokens_from_replacements(placeholder, &replacements).unwrap_or(vec![token]) } else { vec![token] } }) .collect::>(); let statement = Parser::new(dialect.as_ref()) .with_tokens(replaced_tokens) .parse_statements() .map_err(|err| DataFusionError::External(Box::new(err)))?; if statement.len() != 1 { return exec_err!("placeholder replacement should return exactly one statement"); } Ok(statement[0].to_string()) } apache-datafusion-python-54.0.0/crates/core/src/store.rs000066400000000000000000000212611521727353300232070ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use object_store::aws::{AmazonS3, AmazonS3Builder}; use object_store::azure::{MicrosoftAzure, MicrosoftAzureBuilder}; use object_store::gcp::{GoogleCloudStorage, GoogleCloudStorageBuilder}; use object_store::http::{HttpBuilder, HttpStore}; use object_store::local::LocalFileSystem; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; use url::Url; #[derive(FromPyObject)] pub enum StorageContexts { AmazonS3(PyAmazonS3Context), GoogleCloudStorage(PyGoogleCloudContext), MicrosoftAzure(PyMicrosoftAzureContext), LocalFileSystem(PyLocalFileSystemContext), HTTP(PyHttpContext), } #[pyclass( from_py_object, frozen, name = "LocalFileSystem", module = "datafusion.store", subclass )] #[derive(Debug, Clone)] pub struct PyLocalFileSystemContext { pub inner: Arc, } #[pymethods] impl PyLocalFileSystemContext { #[pyo3(signature = (prefix=None))] #[new] fn new(prefix: Option) -> Self { if let Some(prefix) = prefix { Self { inner: Arc::new( LocalFileSystem::new_with_prefix(prefix) .expect("Could not create local LocalFileSystem"), ), } } else { Self { inner: Arc::new(LocalFileSystem::new()), } } } } #[pyclass( from_py_object, frozen, name = "MicrosoftAzure", module = "datafusion.store", subclass )] #[derive(Debug, Clone)] pub struct PyMicrosoftAzureContext { pub inner: Arc, pub container_name: String, } #[pymethods] impl PyMicrosoftAzureContext { #[allow(clippy::too_many_arguments)] #[pyo3(signature = (container_name, account=None, access_key=None, bearer_token=None, client_id=None, client_secret=None, tenant_id=None, sas_query_pairs=None, use_emulator=None, allow_http=None, use_fabric_endpoint=None))] #[new] fn new( container_name: String, account: Option, access_key: Option, bearer_token: Option, client_id: Option, client_secret: Option, tenant_id: Option, sas_query_pairs: Option>, use_emulator: Option, allow_http: Option, use_fabric_endpoint: Option, ) -> Self { let mut builder = MicrosoftAzureBuilder::from_env().with_container_name(&container_name); if let Some(account) = account { builder = builder.with_account(account); } if let Some(access_key) = access_key { builder = builder.with_access_key(access_key); } if let Some(bearer_token) = bearer_token { builder = builder.with_bearer_token_authorization(bearer_token); } match (client_id, client_secret, tenant_id) { (Some(client_id), Some(client_secret), Some(tenant_id)) => { builder = builder.with_client_secret_authorization(client_id, client_secret, tenant_id); } (None, None, None) => {} _ => { panic!("client_id, client_secret, tenat_id must be all set or all None"); } } if let Some(sas_query_pairs) = sas_query_pairs { builder = builder.with_sas_authorization(sas_query_pairs); } if let Some(use_emulator) = use_emulator { builder = builder.with_use_emulator(use_emulator); } if let Some(allow_http) = allow_http { builder = builder.with_allow_http(allow_http); } if let Some(use_fabric_endpoint) = use_fabric_endpoint { builder = builder.with_use_fabric_endpoint(use_fabric_endpoint); } Self { inner: Arc::new( builder .build() .expect("Could not create Azure Storage context"), //TODO: change these to PyErr ), container_name, } } } #[pyclass( from_py_object, frozen, name = "GoogleCloud", module = "datafusion.store", subclass )] #[derive(Debug, Clone)] pub struct PyGoogleCloudContext { pub inner: Arc, pub bucket_name: String, } #[pymethods] impl PyGoogleCloudContext { #[allow(clippy::too_many_arguments)] #[pyo3(signature = (bucket_name, service_account_path=None))] #[new] fn new(bucket_name: String, service_account_path: Option) -> Self { let mut builder = GoogleCloudStorageBuilder::new().with_bucket_name(&bucket_name); if let Some(credential_path) = service_account_path { builder = builder.with_service_account_path(credential_path); } Self { inner: Arc::new( builder .build() .expect("Could not create Google Cloud Storage"), ), bucket_name, } } } #[pyclass( from_py_object, frozen, name = "AmazonS3", module = "datafusion.store", subclass )] #[derive(Debug, Clone)] pub struct PyAmazonS3Context { pub inner: Arc, pub bucket_name: String, } #[pymethods] impl PyAmazonS3Context { #[allow(clippy::too_many_arguments)] #[pyo3(signature = (bucket_name, region=None, access_key_id=None, secret_access_key=None, session_token=None, endpoint=None, allow_http=false, imdsv1_fallback=false))] #[new] fn new( bucket_name: String, region: Option, access_key_id: Option, secret_access_key: Option, session_token: Option, endpoint: Option, //retry_config: RetryConfig, allow_http: bool, imdsv1_fallback: bool, ) -> Self { // start w/ the options that come directly from the environment let mut builder = AmazonS3Builder::from_env(); if let Some(region) = region { builder = builder.with_region(region); } if let Some(access_key_id) = access_key_id { builder = builder.with_access_key_id(access_key_id); }; if let Some(secret_access_key) = secret_access_key { builder = builder.with_secret_access_key(secret_access_key); }; if let Some(session_token) = session_token { builder = builder.with_token(session_token); } if let Some(endpoint) = endpoint { builder = builder.with_endpoint(endpoint); }; if imdsv1_fallback { builder = builder.with_imdsv1_fallback(); }; let store = builder .with_bucket_name(bucket_name.clone()) //.with_retry_config(retry_config) #TODO: add later .with_allow_http(allow_http) .build() .expect("failed to build AmazonS3"); Self { inner: Arc::new(store), bucket_name, } } } #[pyclass( from_py_object, frozen, name = "Http", module = "datafusion.store", subclass )] #[derive(Debug, Clone)] pub struct PyHttpContext { pub url: String, pub store: Arc, } #[pymethods] impl PyHttpContext { #[new] fn new(url: String) -> PyResult { let store = match Url::parse(url.as_str()) { Ok(url) => HttpBuilder::new() .with_url(url.origin().ascii_serialization()) .build(), Err(_) => HttpBuilder::new().build(), } .map_err(|e| PyValueError::new_err(format!("Error: {:?}", e.to_string())))?; Ok(Self { url, store: Arc::new(store), }) } } pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/substrait.rs000066400000000000000000000133731521727353300241000ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use datafusion_python_util::wait_for_future; use datafusion_substrait::logical_plan::{consumer, producer}; use datafusion_substrait::serializer; use datafusion_substrait::substrait::proto::Plan; use prost::Message; use pyo3::prelude::*; use pyo3::types::PyBytes; use crate::context::PySessionContext; use crate::errors::{PyDataFusionError, PyDataFusionResult, py_datafusion_err, to_datafusion_err}; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Plan", module = "datafusion.substrait", subclass )] #[derive(Debug, Clone)] pub struct PyPlan { pub plan: Plan, } #[pymethods] impl PyPlan { fn encode(&self, py: Python) -> PyResult> { let mut proto_bytes = Vec::::new(); self.plan .encode(&mut proto_bytes) .map_err(PyDataFusionError::EncodeError)?; Ok(PyBytes::new(py, &proto_bytes).into()) } /// Get the JSON representation of the substrait plan fn to_json(&self) -> PyDataFusionResult { let json = serde_json::to_string_pretty(&self.plan).map_err(to_datafusion_err)?; Ok(json) } /// Parse a Substrait Plan from its JSON representation #[staticmethod] fn from_json(json: &str) -> PyDataFusionResult { let plan: Plan = serde_json::from_str(json).map_err(to_datafusion_err)?; Ok(PyPlan { plan }) } } impl From for Plan { fn from(plan: PyPlan) -> Plan { plan.plan } } impl From for PyPlan { fn from(plan: Plan) -> PyPlan { PyPlan { plan } } } /// A PySubstraitSerializer is a representation of a Serializer that is capable of both serializing /// a `LogicalPlan` instance to Substrait Protobuf bytes and also deserialize Substrait Protobuf bytes /// to a valid `LogicalPlan` instance. #[pyclass( from_py_object, frozen, name = "Serde", module = "datafusion.substrait", subclass )] #[derive(Debug, Clone)] pub struct PySubstraitSerializer; #[pymethods] impl PySubstraitSerializer { #[staticmethod] pub fn serialize( sql: &str, ctx: PySessionContext, path: &str, py: Python, ) -> PyDataFusionResult<()> { wait_for_future(py, serializer::serialize(sql, &ctx.ctx, path))??; Ok(()) } #[staticmethod] pub fn serialize_to_plan( sql: &str, ctx: PySessionContext, py: Python, ) -> PyDataFusionResult { PySubstraitSerializer::serialize_bytes(sql, ctx, py).and_then(|proto_bytes| { let proto_bytes = proto_bytes.bind(py).cast::().unwrap(); PySubstraitSerializer::deserialize_bytes(proto_bytes.as_bytes().to_vec(), py) }) } #[staticmethod] pub fn serialize_bytes( sql: &str, ctx: PySessionContext, py: Python, ) -> PyDataFusionResult> { let proto_bytes: Vec = wait_for_future(py, serializer::serialize_bytes(sql, &ctx.ctx))??; Ok(PyBytes::new(py, &proto_bytes).into()) } #[staticmethod] pub fn deserialize(path: &str, py: Python) -> PyDataFusionResult { let plan = wait_for_future(py, serializer::deserialize(path))??; Ok(PyPlan { plan: *plan }) } #[staticmethod] pub fn deserialize_bytes(proto_bytes: Vec, py: Python) -> PyDataFusionResult { let plan = wait_for_future(py, serializer::deserialize_bytes(proto_bytes))??; Ok(PyPlan { plan: *plan }) } } #[pyclass( from_py_object, frozen, name = "Producer", module = "datafusion.substrait", subclass )] #[derive(Debug, Clone)] pub struct PySubstraitProducer; #[pymethods] impl PySubstraitProducer { /// Convert DataFusion LogicalPlan to Substrait Plan #[staticmethod] pub fn to_substrait_plan(plan: PyLogicalPlan, ctx: &PySessionContext) -> PyResult { let session_state = ctx.ctx.state(); match producer::to_substrait_plan(&plan.plan, &session_state) { Ok(plan) => Ok(PyPlan { plan: *plan }), Err(e) => Err(py_datafusion_err(e)), } } } #[pyclass( from_py_object, frozen, name = "Consumer", module = "datafusion.substrait", subclass )] #[derive(Debug, Clone)] pub struct PySubstraitConsumer; #[pymethods] impl PySubstraitConsumer { /// Convert Substrait Plan to DataFusion DataFrame #[staticmethod] pub fn from_substrait_plan( ctx: &PySessionContext, plan: PyPlan, py: Python, ) -> PyDataFusionResult { let session_state = ctx.ctx.state(); let result = consumer::from_substrait_plan(&session_state, &plan.plan); let logical_plan = wait_for_future(py, result)??; Ok(PyLogicalPlan::new(logical_plan)) } } pub fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; m.add_class::()?; m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/core/src/table.rs000066400000000000000000000203321521727353300231400ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use arrow::datatypes::SchemaRef; use arrow::pyarrow::ToPyArrow; use async_trait::async_trait; use datafusion::catalog::{Session, TableProviderFactory}; use datafusion::common::Column; use datafusion::datasource::{TableProvider, TableType}; use datafusion::logical_expr::{ CreateExternalTable, Expr, LogicalPlanBuilder, TableProviderFilterPushDown, }; use datafusion::physical_plan::ExecutionPlan; use datafusion::prelude::DataFrame; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; use datafusion_python_util::{create_logical_extension_capsule, table_provider_from_pycapsule}; use pyo3::IntoPyObjectExt; use pyo3::prelude::*; use crate::context::PySessionContext; use crate::dataframe::PyDataFrame; use crate::dataset::Dataset; use crate::errors; use crate::expr::create_external_table::PyCreateExternalTable; /// This struct is used as a common method for all TableProviders, /// whether they refer to an FFI provider, an internally known /// implementation, a dataset, or a dataframe view. #[pyclass( from_py_object, frozen, name = "RawTable", module = "datafusion.catalog", subclass )] #[derive(Clone)] pub struct PyTable { pub table: Arc, } impl PyTable { pub fn table(&self) -> Arc { self.table.clone() } } #[pymethods] impl PyTable { /// Instantiate from any Python object that supports any of the table /// types. We do not know a priori when using this method if the object /// will be passed a wrapped or raw class. Here we handle all of the /// following object types: /// /// - PyTable (essentially a clone operation), but either raw or wrapped /// - DataFrame, either raw or wrapped /// - FFI Table Providers via PyCapsule /// - PyArrow Dataset objects #[new] pub fn new(obj: Bound<'_, PyAny>, session: Option>) -> PyResult { let py = obj.py(); if let Ok(py_table) = obj.extract::() { Ok(py_table) } else if let Ok(py_table) = obj .getattr("_inner") .and_then(|inner| inner.extract::().map_err(Into::::into)) { Ok(py_table) } else if let Ok(py_df) = obj.extract::() { let provider = py_df.inner_df().as_ref().clone().into_view(); Ok(PyTable::from(provider)) } else if let Ok(py_df) = obj .getattr("df") .and_then(|inner| inner.extract::().map_err(Into::::into)) { let provider = py_df.inner_df().as_ref().clone().into_view(); Ok(PyTable::from(provider)) } else if let Some(provider) = { let session = match session { Some(session) => session, None => PySessionContext::global_ctx()?.into_bound_py_any(obj.py())?, }; table_provider_from_pycapsule(obj.clone(), session)? } { Ok(PyTable::from(provider)) } else { let provider = Arc::new(Dataset::new(&obj, py)?) as Arc; Ok(PyTable::from(provider)) } } /// Get a reference to the schema for this table #[getter] fn schema<'py>(&self, py: Python<'py>) -> PyResult> { self.table.schema().to_pyarrow(py) } /// Get the type of this table for metadata/catalog purposes. #[getter] fn kind(&self) -> &str { match self.table.table_type() { TableType::Base => "physical", TableType::View => "view", TableType::Temporary => "temporary", } } fn __repr__(&self) -> PyResult { let kind = self.kind(); Ok(format!("Table(kind={kind})")) } } impl From> for PyTable { fn from(table: Arc) -> Self { Self { table } } } #[derive(Clone, Debug)] pub(crate) struct TempViewTable { df: Arc, } /// This is nearly identical to `DataFrameTableProvider` /// except that it is for temporary tables. /// Remove when https://github.com/apache/datafusion/issues/18026 /// closes. impl TempViewTable { pub(crate) fn new(df: Arc) -> Self { Self { df } } } #[async_trait] impl TableProvider for TempViewTable { fn schema(&self) -> SchemaRef { Arc::new(self.df.schema().as_arrow().clone()) } fn table_type(&self) -> TableType { TableType::Temporary } async fn scan( &self, state: &dyn Session, projection: Option<&Vec>, filters: &[Expr], limit: Option, ) -> datafusion::common::Result> { let filter = filters.iter().cloned().reduce(|acc, new| acc.and(new)); let plan = self.df.logical_plan().clone(); let mut plan = LogicalPlanBuilder::from(plan); if let Some(filter) = filter { plan = plan.filter(filter)?; } let mut plan = if let Some(projection) = projection { // avoiding adding a redundant projection (e.g. SELECT * FROM view) let current_projection = (0..plan.schema().fields().len()).collect::>(); if projection == ¤t_projection { plan } else { let fields: Vec = projection .iter() .map(|i| { Expr::Column(Column::from( self.df.logical_plan().schema().qualified_field(*i), )) }) .collect(); plan.project(fields)? } } else { plan }; if let Some(limit) = limit { plan = plan.limit(0, Some(limit))?; } state.create_physical_plan(&plan.build()?).await } fn supports_filters_pushdown( &self, filters: &[&Expr], ) -> datafusion::common::Result> { Ok(vec![TableProviderFilterPushDown::Exact; filters.len()]) } } #[derive(Debug)] pub(crate) struct RustWrappedPyTableProviderFactory { pub(crate) table_provider_factory: Py, pub(crate) codec: Arc, } impl RustWrappedPyTableProviderFactory { pub fn new(table_provider_factory: Py, codec: Arc) -> Self { Self { table_provider_factory, codec, } } fn create_inner( &self, cmd: CreateExternalTable, codec: Bound, ) -> PyResult> { Python::attach(|py| { let provider = self.table_provider_factory.bind(py); let cmd = PyCreateExternalTable::from(cmd); provider .call_method1("create", (cmd,)) .and_then(|t| PyTable::new(t, Some(codec))) .map(|t| t.table()) }) } } #[async_trait] impl TableProviderFactory for RustWrappedPyTableProviderFactory { async fn create( &self, _: &dyn Session, cmd: &CreateExternalTable, ) -> datafusion::common::Result> { Python::attach(|py| { let codec = create_logical_extension_capsule(py, self.codec.as_ref()) .map_err(errors::to_datafusion_err)?; self.create_inner(cmd.clone(), codec.into_any()) .map_err(errors::to_datafusion_err) }) } } apache-datafusion-python-54.0.0/crates/core/src/udaf.rs000066400000000000000000000321751521727353300230000ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ptr::NonNull; use std::sync::Arc; use datafusion::arrow::array::ArrayRef; use datafusion::arrow::datatypes::{DataType, Field, FieldRef}; use datafusion::arrow::pyarrow::{PyArrowType, ToPyArrow}; use datafusion::common::ScalarValue; use datafusion::error::{DataFusionError, Result}; use datafusion::logical_expr::function::{AccumulatorArgs, StateFieldsArgs}; use datafusion::logical_expr::{ Accumulator, AccumulatorFactoryFunction, AggregateUDF, AggregateUDFImpl, Signature, Volatility, }; use datafusion_ffi::udaf::FFI_AggregateUDF; use datafusion_python_util::parse_volatility; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyTuple}; use crate::common::data_type::PyScalarValue; use crate::errors::{PyDataFusionResult, py_datafusion_err, to_datafusion_err}; use crate::expr::PyExpr; #[derive(Debug)] struct RustAccumulator { accum: Py, } impl RustAccumulator { fn new(accum: Py) -> Self { Self { accum } } } impl Accumulator for RustAccumulator { fn state(&mut self) -> Result> { Python::attach(|py| -> PyResult> { let values = self.accum.bind(py).call_method0("state")?; let mut scalars = Vec::new(); for item in values.try_iter()? { let item: Bound<'_, PyAny> = item?; let scalar = item.extract::()?.0; scalars.push(scalar); } Ok(scalars) }) .map_err(|e| DataFusionError::Execution(format!("{e}"))) } fn evaluate(&mut self) -> Result { Python::attach(|py| -> PyResult { let value = self.accum.bind(py).call_method0("evaluate")?; value.extract::().map(|v| v.0) }) .map_err(|e| DataFusionError::Execution(format!("{e}"))) } fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> { Python::attach(|py| { // 1. cast args to Pyarrow array let py_args = values .iter() .map(|arg| arg.to_data().to_pyarrow(py).unwrap()) .collect::>(); let py_args = PyTuple::new(py, py_args).map_err(to_datafusion_err)?; // 2. call function self.accum .bind(py) .call_method1("update", py_args) .map_err(|e| DataFusionError::Execution(format!("{e}")))?; Ok(()) }) } fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> { Python::attach(|py| { // // 1. cast states to Pyarrow arrays let py_states: Result>> = states .iter() .map(|state| { state .to_data() .to_pyarrow(py) .map_err(|e| DataFusionError::Execution(format!("{e}"))) }) .collect(); // 2. call merge self.accum .bind(py) .call_method1("merge", (py_states?,)) .map_err(|e| DataFusionError::Execution(format!("{e}")))?; Ok(()) }) } fn size(&self) -> usize { std::mem::size_of_val(self) } fn retract_batch(&mut self, values: &[ArrayRef]) -> Result<()> { Python::attach(|py| { // 1. cast args to Pyarrow array let py_args = values .iter() .map(|arg| arg.to_data().to_pyarrow(py).unwrap()) .collect::>(); let py_args = PyTuple::new(py, py_args).map_err(to_datafusion_err)?; // 2. call function self.accum .bind(py) .call_method1("retract_batch", py_args) .map_err(|e| DataFusionError::Execution(format!("{e}")))?; Ok(()) }) } fn supports_retract_batch(&self) -> bool { Python::attach( |py| match self.accum.bind(py).call_method0("supports_retract_batch") { Ok(x) => x.extract().unwrap_or(false), Err(_) => false, }, ) } } fn instantiate_accumulator(accum: &Py) -> Result> { let instance = Python::attach(|py| { accum .call0(py) .map_err(|e| DataFusionError::Execution(format!("{e}"))) })?; Ok(Box::new(RustAccumulator::new(instance))) } /// Wrap a Python accumulator factory in an `AccumulatorFactoryFunction`. /// /// Retained for downstream callers that previously consumed this /// helper to build a [`AccumulatorFactoryFunction`] for `create_udaf` /// or similar factory-based APIs. New in-crate code should construct /// a [`PythonFunctionAggregateUDF`] directly so the codec can downcast /// and ship it inline. pub fn to_rust_accumulator(accum: Py) -> AccumulatorFactoryFunction { Arc::new(move |_args| instantiate_accumulator(&accum)) } /// Named-struct `AggregateUDFImpl` for Python-defined aggregate UDFs. /// Holds the Python accumulator factory directly so the codec can /// downcast and cloudpickle it across process boundaries. #[derive(Debug)] pub(crate) struct PythonFunctionAggregateUDF { name: String, accumulator: Py, signature: Signature, return_type: DataType, state_fields: Vec, } impl PythonFunctionAggregateUDF { fn new( name: String, accumulator: Py, input_types: Vec, return_type: DataType, state_types: Vec, volatility: Volatility, ) -> Self { let signature = Signature::exact(input_types, volatility); let state_fields = state_types .into_iter() .enumerate() .map(|(i, t)| Arc::new(Field::new(format!("state_{i}"), t, true))) .collect(); Self { name, accumulator, signature, return_type, state_fields, } } /// Stored Python callable that returns a fresh accumulator instance /// per partition. Consumed by the codec to cloudpickle the factory /// across process boundaries. pub(crate) fn accumulator(&self) -> &Py { &self.accumulator } pub(crate) fn return_type(&self) -> &DataType { &self.return_type } pub(crate) fn state_fields_ref(&self) -> &[FieldRef] { &self.state_fields } /// Reconstruct a `PythonFunctionAggregateUDF` from the parts emitted /// by the codec. `state_fields` carries the full state schema /// (names, data types, nullability, metadata) — the codec extracts /// it from the IPC payload, so the post-decode state schema is /// identical to the pre-encode one. Use [`Self::new`] when only /// `Vec` is available (e.g. the Python constructor path, /// where field names are synthesized). pub(crate) fn from_parts( name: String, accumulator: Py, input_types: Vec, return_type: DataType, state_fields: Vec, volatility: Volatility, ) -> Self { Self { name, accumulator, signature: Signature::exact(input_types, volatility), return_type, state_fields, } } } impl Eq for PythonFunctionAggregateUDF {} impl PartialEq for PythonFunctionAggregateUDF { fn eq(&self, other: &Self) -> bool { self.name == other.name && self.signature == other.signature && self.return_type == other.return_type && self.state_fields == other.state_fields // Pointer-identity fast path: `Arc`-shared clones of the // same UDF skip the GIL roundtrip. Falls through to Python // `__eq__` only for two distinct callables. && (self.accumulator.as_ptr() == other.accumulator.as_ptr() || Python::attach(|py| { // See `PythonFunctionScalarUDF::eq` for the // rationale on swallowing the exception as `false` // and logging at `debug`. FIXME: revisit if // upstream `AggregateUDFImpl` exposes a fallible // `PartialEq`. self.accumulator .bind(py) .eq(other.accumulator.bind(py)) .unwrap_or_else(|e| { log::debug!( target: "datafusion_python::udaf", "PythonFunctionAggregateUDF {:?} __eq__ raised; treating as unequal: {e}", self.name, ); false }) })) } } impl std::hash::Hash for PythonFunctionAggregateUDF { fn hash(&self, state: &mut H) { // See `PythonFunctionScalarUDF`'s `Hash` impl for the // rationale: hash the identifying header only and let // `PartialEq` disambiguate callables. self.name.hash(state); self.signature.hash(state); self.return_type.hash(state); for f in &self.state_fields { f.hash(state); } } } impl AggregateUDFImpl for PythonFunctionAggregateUDF { fn name(&self) -> &str { &self.name } fn signature(&self) -> &Signature { &self.signature } fn return_type(&self, _arg_types: &[DataType]) -> Result { Ok(self.return_type.clone()) } fn accumulator(&self, _acc_args: AccumulatorArgs) -> Result> { instantiate_accumulator(&self.accumulator) } fn state_fields(&self, _args: StateFieldsArgs) -> Result> { Ok(self.state_fields.clone()) } } fn aggregate_udf_from_capsule(capsule: &Bound<'_, PyCapsule>) -> PyDataFusionResult { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_aggregate_udf"))? .cast(); let udaf = unsafe { data.as_ref() }; let udaf: Arc = udaf.into(); Ok(AggregateUDF::new_from_shared_impl(udaf)) } /// Represents an AggregateUDF #[pyclass( from_py_object, frozen, name = "AggregateUDF", module = "datafusion", subclass )] #[derive(Debug, Clone)] pub struct PyAggregateUDF { pub(crate) function: AggregateUDF, } #[pymethods] impl PyAggregateUDF { #[new] #[pyo3(signature=(name, accumulator, input_type, return_type, state_type, volatility))] fn new( name: &str, accumulator: Py, input_type: PyArrowType>, return_type: PyArrowType, state_type: PyArrowType>, volatility: &str, ) -> PyResult { let py_udf = PythonFunctionAggregateUDF::new( name.to_string(), accumulator, input_type.0, return_type.0, state_type.0, parse_volatility(volatility)?, ); let function = AggregateUDF::new_from_impl(py_udf); Ok(Self { function }) } #[staticmethod] pub fn from_pycapsule(func: Bound<'_, PyAny>) -> PyDataFusionResult { if func.is_instance_of::() { let capsule = func.cast::().map_err(py_datafusion_err)?; let function = aggregate_udf_from_capsule(capsule)?; return Ok(Self { function }); } if func.hasattr("__datafusion_aggregate_udf__")? { let capsule = func.getattr("__datafusion_aggregate_udf__")?.call0()?; let capsule = capsule.cast::().map_err(py_datafusion_err)?; let function = aggregate_udf_from_capsule(capsule)?; return Ok(Self { function }); } Err(crate::errors::PyDataFusionError::Common( "__datafusion_aggregate_udf__ does not exist on AggregateUDF object.".to_string(), )) } /// creates a new PyExpr with the call of the udf #[pyo3(signature = (*args))] fn __call__(&self, args: Vec) -> PyResult { let args = args.iter().map(|e| e.expr.clone()).collect(); Ok(self.function.call(args).into()) } fn __repr__(&self) -> PyResult { Ok(format!("AggregateUDF({})", self.function.name())) } #[getter] fn name(&self) -> &str { self.function.name() } } apache-datafusion-python-54.0.0/crates/core/src/udf.rs000066400000000000000000000234501521727353300226330ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::hash::{Hash, Hasher}; use std::ptr::NonNull; use std::sync::Arc; use arrow::datatypes::{Field, FieldRef}; use arrow::pyarrow::ToPyArrow; use datafusion::arrow::array::{ArrayData, make_array}; use datafusion::arrow::datatypes::DataType; use datafusion::arrow::pyarrow::{FromPyArrow, PyArrowType}; use datafusion::common::internal_err; use datafusion::error::DataFusionError; use datafusion::logical_expr::{ ColumnarValue, ReturnFieldArgs, ScalarFunctionArgs, ScalarUDF, ScalarUDFImpl, Signature, Volatility, }; use datafusion_ffi::udf::FFI_ScalarUDF; use datafusion_python_util::parse_volatility; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyTuple}; use crate::array::PyArrowArrayExportable; use crate::errors::{PyDataFusionResult, to_datafusion_err}; use crate::expr::PyExpr; /// This struct holds the Python written function that is a /// ScalarUDF. #[derive(Debug)] pub(crate) struct PythonFunctionScalarUDF { name: String, func: Py, signature: Signature, return_field: FieldRef, } impl PythonFunctionScalarUDF { fn new( name: String, func: Py, input_fields: Vec, return_field: Field, volatility: Volatility, ) -> Self { let input_types = input_fields.iter().map(|f| f.data_type().clone()).collect(); let signature = Signature::exact(input_types, volatility); Self { name, func, signature, return_field: Arc::new(return_field), } } /// Stored Python callable. Consumed by the codec to cloudpickle /// the function body across process boundaries. pub(crate) fn func(&self) -> &Py { &self.func } pub(crate) fn return_field(&self) -> &FieldRef { &self.return_field } /// Reconstruct a `PythonFunctionScalarUDF` from the parts emitted /// by the codec. Inputs collapse to `Vec` because /// `Signature::exact` cannot carry per-input nullability or /// metadata — the encoder is free to discard that side of the /// schema. `return_field` is kept as a `Field` so the post-decode /// nullability and metadata match the sender's instance. pub(crate) fn from_parts( name: String, func: Py, input_types: Vec, return_field: Field, volatility: Volatility, ) -> Self { Self { name, func, signature: Signature::exact(input_types, volatility), return_field: Arc::new(return_field), } } } impl Eq for PythonFunctionScalarUDF {} impl PartialEq for PythonFunctionScalarUDF { fn eq(&self, other: &Self) -> bool { self.name == other.name && self.signature == other.signature && self.return_field == other.return_field // Identical pointers ⇒ same Python object. Most equality // checks compare `Arc`-shared clones of the same UDF // (e.g. expression rewriting), so the pointer match short- // circuits before touching the GIL. && (self.func.as_ptr() == other.func.as_ptr() || Python::attach(|py| { // Rust's `PartialEq` cannot return `Result`, so we // have to pick a side when Python `__eq__` raises. // `false` is the conservative choice — better to // report two UDFs as distinct than to wrongly // merge them — but the silent miss can still // surface as expression-dedup or cache-lookup // anomalies. Log at `debug` so the failure is // observable without flooding production logs. // FIXME: revisit if upstream `ScalarUDFImpl` // exposes a fallible `PartialEq`. self.func .bind(py) .eq(other.func.bind(py)) .unwrap_or_else(|e| { log::debug!( target: "datafusion_python::udf", "PythonFunctionScalarUDF {:?} __eq__ raised; treating as unequal: {e}", self.name, ); false }) })) } } impl Hash for PythonFunctionScalarUDF { fn hash(&self, state: &mut H) { // Hash only the identifying header (name + signature + return // field). Skipping `func` is intentional: the Rust `Hash` // contract requires `a == b ⇒ hash(a) == hash(b)`, not the // converse, so a coarser hash is sound — `PartialEq` still // disambiguates two UDFs with the same header but distinct // callables. Falling back to a sentinel on `py_hash` failure // (as a prior revision did) silently mapped every unhashable // closure to the same bucket; that is the worst case for a // hashmap and is what this rewrite avoids. self.name.hash(state); self.signature.hash(state); self.return_field.hash(state); } } impl ScalarUDFImpl for PythonFunctionScalarUDF { fn name(&self) -> &str { &self.name } fn signature(&self) -> &Signature { &self.signature } fn return_type(&self, _arg_types: &[DataType]) -> datafusion::common::Result { internal_err!( "return_field should not be called when return_field_from_args is implemented." ) } fn return_field_from_args( &self, _args: ReturnFieldArgs, ) -> datafusion::common::Result { Ok(Arc::clone(&self.return_field)) } fn invoke_with_args( &self, args: ScalarFunctionArgs, ) -> datafusion::common::Result { let num_rows = args.number_rows; Python::attach(|py| { // 1. cast args to Pyarrow arrays let py_args = args .args .into_iter() .zip(args.arg_fields) .map(|(arg, field)| { let array = arg.to_array(num_rows)?; PyArrowArrayExportable::new(array, field) .to_pyarrow(py) .map_err(to_datafusion_err) }) .collect::, _>>()?; let py_args = PyTuple::new(py, py_args).map_err(to_datafusion_err)?; // 2. call function let value = self .func .call(py, py_args, None) .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?; // 3. cast to arrow::array::Array let array_data = ArrayData::from_pyarrow_bound(value.bind(py)) .map_err(|e| DataFusionError::Execution(format!("{e:?}")))?; Ok(ColumnarValue::Array(make_array(array_data))) }) } } /// Represents a PyScalarUDF #[pyclass( from_py_object, frozen, name = "ScalarUDF", module = "datafusion", subclass )] #[derive(Debug, Clone)] pub struct PyScalarUDF { pub(crate) function: ScalarUDF, } #[pymethods] impl PyScalarUDF { #[new] #[pyo3(signature=(name, func, input_types, return_type, volatility))] fn new( name: String, func: Py, input_types: PyArrowType>, return_type: PyArrowType, volatility: &str, ) -> PyResult { let py_function = PythonFunctionScalarUDF::new( name, func, input_types.0, return_type.0, parse_volatility(volatility)?, ); let function = ScalarUDF::new_from_impl(py_function); Ok(Self { function }) } #[staticmethod] pub fn from_pycapsule(func: Bound<'_, PyAny>) -> PyDataFusionResult { if func.hasattr("__datafusion_scalar_udf__")? { let capsule = func.getattr("__datafusion_scalar_udf__")?.call0()?; let capsule = capsule.cast::().map_err(to_datafusion_err)?; let data: NonNull = capsule .pointer_checked(Some(c"datafusion_scalar_udf"))? .cast(); let udf = unsafe { data.as_ref() }; let udf: Arc = udf.into(); Ok(Self { function: ScalarUDF::new_from_shared_impl(udf), }) } else { Err(crate::errors::PyDataFusionError::Common( "__datafusion_scalar_udf__ does not exist on ScalarUDF object.".to_string(), )) } } /// creates a new PyExpr with the call of the udf #[pyo3(signature = (*args))] fn __call__(&self, args: Vec) -> PyResult { let args = args.iter().map(|e| e.expr.clone()).collect(); Ok(self.function.call(args).into()) } fn __repr__(&self) -> PyResult { Ok(format!("ScalarUDF({})", self.function.name())) } #[getter] fn name(&self) -> &str { self.function.name() } } apache-datafusion-python-54.0.0/crates/core/src/udtf.rs000066400000000000000000000172171521727353300230230ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ptr::NonNull; use std::sync::Arc; use datafusion::catalog::{Session, TableFunctionArgs, TableFunctionImpl, TableProvider}; use datafusion::error::{DataFusionError, Result as DataFusionResult}; use datafusion::execution::context::SessionContext; use datafusion::execution::session_state::SessionState; use datafusion::logical_expr::Expr; use datafusion_ffi::udtf::FFI_TableFunction; use pyo3::IntoPyObjectExt; use pyo3::exceptions::{PyImportError, PyTypeError}; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyDict, PyTuple, PyType}; use crate::context::PySessionContext; use crate::errors::{py_datafusion_err, to_datafusion_err}; use crate::expr::PyExpr; use crate::table::PyTable; /// A pure-Python UDTF callable plus the metadata we discovered about it /// at registration time. #[derive(Debug, Clone)] pub(crate) struct PythonTableFunctionCallable { pub(crate) callable: Arc>, /// When true, the calling :class:`SessionContext` is passed to the /// callable as a ``session`` keyword argument on every invocation. /// Opt-in at registration time via ``with_session=True`` on the /// Python wrapper. pub(crate) inject_session_on_call: bool, } /// Represents a user defined table function #[pyclass(from_py_object, frozen, name = "TableFunction", module = "datafusion")] #[derive(Debug, Clone)] pub struct PyTableFunction { pub(crate) name: String, pub(crate) inner: PyTableFunctionInner, } #[derive(Debug, Clone)] pub(crate) enum PyTableFunctionInner { PythonFunction(PythonTableFunctionCallable), FFIFunction(Arc), } #[pymethods] impl PyTableFunction { #[new] #[pyo3(signature=(name, func, session, inject_session_on_call=false))] pub fn new( name: &str, func: Bound<'_, PyAny>, session: Option>, inject_session_on_call: bool, ) -> PyResult { let inner = if func.hasattr("__datafusion_table_function__")? { let py = func.py(); let session = match session { Some(session) => session, None => PySessionContext::global_ctx()?.into_bound_py_any(py)?, }; let capsule = func .getattr("__datafusion_table_function__")? .call1((session,)).map_err(|err| { if err.get_type(py).is(PyType::new::(py)) { PyImportError::new_err("Incompatible libraries. DataFusion 52.0.0 introduced an incompatible signature change for table functions. Either downgrade DataFusion or upgrade your function library.") } else { err } })?; let capsule = capsule.cast::()?; let data: NonNull = capsule .pointer_checked(Some(c"datafusion_table_function"))? .cast(); let ffi_func = unsafe { data.as_ref() }; let foreign_func: Arc = ffi_func.to_owned().into(); PyTableFunctionInner::FFIFunction(foreign_func) } else { PyTableFunctionInner::PythonFunction(PythonTableFunctionCallable { callable: Arc::new(func.unbind()), inject_session_on_call, }) }; Ok(Self { name: name.to_string(), inner, }) } #[pyo3(signature = (*args))] pub fn __call__(&self, args: Vec) -> PyResult { let args: Vec = args.iter().map(|e| e.expr.clone()).collect(); let global = PySessionContext::global_ctx()?; let state = global.ctx.state(); let table_provider = self .call_with_args(TableFunctionArgs::new(&args, &state)) .map_err(py_datafusion_err)?; Ok(PyTable::from(table_provider)) } fn __repr__(&self) -> PyResult { Ok(format!("TableUDF({})", self.name)) } } /// Materialize a fresh :class:`PySessionContext` from the borrowed /// ``&dyn Session`` handed in at call time. /// /// Upstream invokes ``call_with_args`` with a trait-object reference /// rather than an owned context; we downcast it to the canonical /// :class:`SessionState` impl and rebuild a :class:`SessionContext` /// (sharing the same registries via the Arc-heavy interior of /// :class:`SessionState`). /// /// The downcast is defensive. Every path that reaches a pure-Python /// UDTF today hands us a `SessionState`: the SQL planner builds the /// args from its own `SessionState`, and `PyTableFunction::__call__` /// uses the global context's state. A non-`SessionState` session /// (e.g. a `ForeignSession`) would only arrive if this UDTF were /// exported across the FFI boundary to a foreign-library consumer, /// which datafusion-python does not do. Should that change, this /// returns an error rather than silently misbehaving. fn py_session_from_session(session: &dyn Session) -> DataFusionResult { let state = session .as_any() .downcast_ref::() .ok_or_else(|| { DataFusionError::Execution( "Cannot expose this UDTF's calling session to Python: the \ session is not a SessionState. Drop the `session` keyword \ from the callback signature to fall back to the \ expression-only call form." .to_string(), ) })?; Ok(PySessionContext::from(SessionContext::new_with_state( state.clone(), ))) } #[allow(clippy::result_large_err)] fn call_python_table_function( func: &PythonTableFunctionCallable, args: TableFunctionArgs, ) -> DataFusionResult> { let py_session = if func.inject_session_on_call { Some(py_session_from_session(args.session())?) } else { None }; let py_exprs = args .exprs() .iter() .map(|arg| PyExpr::from(arg.clone())) .collect::>(); Python::attach(|py| { let py_args = PyTuple::new(py, py_exprs)?; let provider_obj = if let Some(session) = py_session { let kwargs = PyDict::new(py); kwargs.set_item("session", session.into_pyobject(py)?)?; func.callable.call(py, py_args, Some(&kwargs))? } else { func.callable.call1(py, py_args)? }; let provider = provider_obj.bind(py).clone(); Ok::, PyErr>(PyTable::new(provider, None)?.table) }) .map_err(to_datafusion_err) } impl TableFunctionImpl for PyTableFunction { fn call_with_args(&self, args: TableFunctionArgs) -> DataFusionResult> { match &self.inner { PyTableFunctionInner::FFIFunction(func) => func.call_with_args(args), PyTableFunctionInner::PythonFunction(callable) => { call_python_table_function(callable, args) } } } } apache-datafusion-python-54.0.0/crates/core/src/udwf.rs000066400000000000000000000327311521727353300230240ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::ops::Range; use std::ptr::NonNull; use std::sync::Arc; use arrow::array::{Array, ArrayData, ArrayRef, make_array}; use datafusion::arrow::datatypes::DataType; use datafusion::arrow::pyarrow::{FromPyArrow, PyArrowType, ToPyArrow}; use datafusion::error::{DataFusionError, Result}; use datafusion::logical_expr::function::{PartitionEvaluatorArgs, WindowUDFFieldArgs}; use datafusion::logical_expr::window_state::WindowAggState; use datafusion::logical_expr::{ PartitionEvaluator, PartitionEvaluatorFactory, Signature, Volatility, WindowUDF, WindowUDFImpl, }; use datafusion::scalar::ScalarValue; use datafusion_ffi::udwf::FFI_WindowUDF; use datafusion_python_util::parse_volatility; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyList, PyTuple}; use crate::common::data_type::PyScalarValue; use crate::errors::{PyDataFusionResult, to_datafusion_err}; use crate::expr::PyExpr; #[derive(Debug)] struct RustPartitionEvaluator { evaluator: Py, } impl RustPartitionEvaluator { fn new(evaluator: Py) -> Self { Self { evaluator } } } impl PartitionEvaluator for RustPartitionEvaluator { fn memoize(&mut self, _state: &mut WindowAggState) -> Result<()> { Python::attach(|py| self.evaluator.bind(py).call_method0("memoize").map(|_| ())) .map_err(|e| DataFusionError::Execution(format!("{e}"))) } fn get_range(&self, idx: usize, n_rows: usize) -> Result> { Python::attach(|py| { let py_args = vec![idx.into_pyobject(py)?, n_rows.into_pyobject(py)?]; let py_args = PyTuple::new(py, py_args)?; self.evaluator .bind(py) .call_method1("get_range", py_args) .and_then(|v| { let tuple: Bound<'_, PyTuple> = v.extract()?; if tuple.len() != 2 { return Err(PyValueError::new_err(format!( "Expected get_range to return tuple of length 2. Received length {}", tuple.len() ))); } let start: usize = tuple.get_item(0).unwrap().extract()?; let end: usize = tuple.get_item(1).unwrap().extract()?; Ok(Range { start, end }) }) }) .map_err(|e| DataFusionError::Execution(format!("{e}"))) } fn is_causal(&self) -> bool { Python::attach(|py| { self.evaluator .bind(py) .call_method0("is_causal") .and_then(|v| v.extract()) .unwrap_or(false) }) } fn evaluate_all(&mut self, values: &[ArrayRef], num_rows: usize) -> Result { Python::attach(|py| { let py_values = PyList::new( py, values .iter() .map(|arg| arg.into_data().to_pyarrow(py).unwrap()), )?; let py_num_rows = num_rows.into_pyobject(py)?; let py_args = PyTuple::new(py, vec![py_values.as_any(), &py_num_rows])?; self.evaluator .bind(py) .call_method1("evaluate_all", py_args) .map(|v| { let array_data = ArrayData::from_pyarrow_bound(&v).unwrap(); make_array(array_data) }) }) .map_err(to_datafusion_err) } fn evaluate(&mut self, values: &[ArrayRef], range: &Range) -> Result { Python::attach(|py| { let py_values = PyList::new( py, values .iter() .map(|arg| arg.into_data().to_pyarrow(py).unwrap()), )?; let range_tuple = PyTuple::new(py, vec![range.start, range.end])?; let py_args = PyTuple::new(py, vec![py_values.as_any(), range_tuple.as_any()])?; self.evaluator .bind(py) .call_method1("evaluate", py_args) .and_then(|v| v.extract::()) .map(|v| v.0) }) .map_err(to_datafusion_err) } fn evaluate_all_with_rank( &self, num_rows: usize, ranks_in_partition: &[Range], ) -> Result { Python::attach(|py| { let ranks = ranks_in_partition .iter() .map(|r| PyTuple::new(py, vec![r.start, r.end])) .collect::>>()?; // 1. cast args to Pyarrow array let py_args = vec![ num_rows.into_pyobject(py)?.into_any(), PyList::new(py, ranks)?.into_any(), ]; let py_args = PyTuple::new(py, py_args)?; // 2. call function self.evaluator .bind(py) .call_method1("evaluate_all_with_rank", py_args) .map(|v| { let array_data = ArrayData::from_pyarrow_bound(&v).unwrap(); make_array(array_data) }) }) .map_err(to_datafusion_err) } fn supports_bounded_execution(&self) -> bool { Python::attach(|py| { self.evaluator .bind(py) .call_method0("supports_bounded_execution") .and_then(|v| v.extract()) .unwrap_or(false) }) } fn uses_window_frame(&self) -> bool { Python::attach(|py| { self.evaluator .bind(py) .call_method0("uses_window_frame") .and_then(|v| v.extract()) .unwrap_or(false) }) } fn include_rank(&self) -> bool { Python::attach(|py| { self.evaluator .bind(py) .call_method0("include_rank") .and_then(|v| v.extract()) .unwrap_or(false) }) } } fn instantiate_partition_evaluator(evaluator: &Py) -> Result> { let instance = Python::attach(|py| { evaluator .call0(py) .map_err(|e| DataFusionError::Execution(e.to_string())) })?; Ok(Box::new(RustPartitionEvaluator::new(instance))) } /// Wrap a Python evaluator factory in a `PartitionEvaluatorFactory`. /// /// Retained for downstream callers that previously consumed this /// helper to build a [`PartitionEvaluatorFactory`] for factory-based /// APIs. New in-crate code should construct a /// [`PythonFunctionWindowUDF`] directly so the codec can downcast and /// ship it inline. pub fn to_rust_partition_evaluator(evaluator: Py) -> PartitionEvaluatorFactory { Arc::new(move || instantiate_partition_evaluator(&evaluator)) } /// Represents an WindowUDF #[pyclass( from_py_object, frozen, name = "WindowUDF", module = "datafusion", subclass )] #[derive(Debug, Clone)] pub struct PyWindowUDF { pub(crate) function: WindowUDF, } #[pymethods] impl PyWindowUDF { #[new] #[pyo3(signature=(name, evaluator, input_types, return_type, volatility))] fn new( name: &str, evaluator: Py, input_types: Vec>, return_type: PyArrowType, volatility: &str, ) -> PyResult { let return_type = return_type.0; let input_types: Vec = input_types.into_iter().map(|t| t.0).collect(); let function = WindowUDF::from(PythonFunctionWindowUDF::new( name, evaluator, input_types, return_type, parse_volatility(volatility)?, )); Ok(Self { function }) } /// creates a new PyExpr with the call of the udf #[pyo3(signature = (*args))] fn __call__(&self, args: Vec) -> PyResult { let args = args.iter().map(|e| e.expr.clone()).collect(); Ok(self.function.call(args).into()) } #[staticmethod] pub fn from_pycapsule(func: Bound<'_, PyAny>) -> PyDataFusionResult { let capsule = if func.hasattr("__datafusion_window_udf__")? { func.getattr("__datafusion_window_udf__")?.call0()? } else { func }; let capsule = capsule.cast::().map_err(to_datafusion_err)?; let data: NonNull = capsule .pointer_checked(Some(c"datafusion_window_udf"))? .cast(); let udwf = unsafe { data.as_ref() }; let udwf: Arc = udwf.into(); Ok(Self { function: WindowUDF::new_from_shared_impl(udwf), }) } fn __repr__(&self) -> PyResult { Ok(format!("WindowUDF({})", self.function.name())) } #[getter] fn name(&self) -> &str { self.function.name() } } /// `WindowUDFImpl` for Python-defined window UDFs. /// /// Holds the Python evaluator factory directly so the codec can /// downcast and cloudpickle it across process boundaries. Replaces /// the prior factory-erased `MultiColumnWindowUDF`; the old name is /// kept as a type alias below for backward compatibility. #[derive(Debug)] pub struct PythonFunctionWindowUDF { name: String, evaluator: Py, signature: Signature, return_type: DataType, } /// Backward-compatible alias for downstream crates that referenced the /// previous struct name. New code should use [`PythonFunctionWindowUDF`]. pub type MultiColumnWindowUDF = PythonFunctionWindowUDF; impl PythonFunctionWindowUDF { pub fn new( name: impl Into, evaluator: Py, input_types: Vec, return_type: DataType, volatility: Volatility, ) -> Self { let name = name.into(); let signature = Signature::exact(input_types, volatility); Self { name, evaluator, signature, return_type, } } /// Stored Python callable that produces a fresh partition /// evaluator instance per partition. Consumed by the codec to /// cloudpickle the evaluator factory across process boundaries. pub(crate) fn evaluator(&self) -> &Py { &self.evaluator } pub(crate) fn return_type(&self) -> &DataType { &self.return_type } } impl Eq for PythonFunctionWindowUDF {} impl PartialEq for PythonFunctionWindowUDF { fn eq(&self, other: &Self) -> bool { self.name == other.name && self.signature == other.signature && self.return_type == other.return_type // Pointer-identity fast path: `Arc`-shared clones of the // same UDF skip the GIL roundtrip. Falls through to Python // `__eq__` only for two distinct callables. && (self.evaluator.as_ptr() == other.evaluator.as_ptr() || Python::attach(|py| { // See `PythonFunctionScalarUDF::eq` for the // rationale on swallowing the exception as `false` // and logging at `debug`. FIXME: revisit if // upstream `WindowUDFImpl` exposes a fallible // `PartialEq`. self.evaluator .bind(py) .eq(other.evaluator.bind(py)) .unwrap_or_else(|e| { log::debug!( target: "datafusion_python::udwf", "PythonFunctionWindowUDF {:?} __eq__ raised; treating as unequal: {e}", self.name, ); false }) })) } } impl std::hash::Hash for PythonFunctionWindowUDF { fn hash(&self, state: &mut H) { // See `PythonFunctionScalarUDF`'s `Hash` impl for the // rationale: hash the identifying header only and let // `PartialEq` disambiguate evaluators. self.name.hash(state); self.signature.hash(state); self.return_type.hash(state); } } impl WindowUDFImpl for PythonFunctionWindowUDF { fn name(&self) -> &str { &self.name } fn signature(&self) -> &Signature { &self.signature } fn field(&self, field_args: WindowUDFFieldArgs) -> Result { // TODO: Should nullable always be `true`? Ok(arrow::datatypes::Field::new(field_args.name(), self.return_type.clone(), true).into()) } // TODO: Enable passing partition_evaluator_args to python? fn partition_evaluator( &self, _partition_evaluator_args: PartitionEvaluatorArgs, ) -> Result> { instantiate_partition_evaluator(&self.evaluator) } } apache-datafusion-python-54.0.0/crates/core/src/unparser/000077500000000000000000000000001521727353300233425ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/core/src/unparser/dialect.rs000066400000000000000000000035161521727353300253220ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::sync::Arc; use datafusion::sql::unparser::dialect::{ DefaultDialect, Dialect, DuckDBDialect, MySqlDialect, PostgreSqlDialect, SqliteDialect, }; use pyo3::prelude::*; #[pyclass( from_py_object, frozen, name = "Dialect", module = "datafusion.unparser", subclass )] #[derive(Clone)] pub struct PyDialect { pub dialect: Arc, } #[pymethods] impl PyDialect { #[staticmethod] pub fn default() -> Self { Self { dialect: Arc::new(DefaultDialect {}), } } #[staticmethod] pub fn postgres() -> Self { Self { dialect: Arc::new(PostgreSqlDialect {}), } } #[staticmethod] pub fn mysql() -> Self { Self { dialect: Arc::new(MySqlDialect {}), } } #[staticmethod] pub fn sqlite() -> Self { Self { dialect: Arc::new(SqliteDialect {}), } } #[staticmethod] pub fn duckdb() -> Self { Self { dialect: Arc::new(DuckDBDialect::new()), } } } apache-datafusion-python-54.0.0/crates/core/src/unparser/mod.rs000066400000000000000000000041011521727353300244630ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. mod dialect; use std::sync::Arc; use datafusion::sql::unparser::Unparser; use datafusion::sql::unparser::dialect::Dialect; use dialect::PyDialect; use pyo3::exceptions::PyValueError; use pyo3::prelude::*; use crate::sql::logical::PyLogicalPlan; #[pyclass( from_py_object, frozen, name = "Unparser", module = "datafusion.unparser", subclass )] #[derive(Clone)] pub struct PyUnparser { dialect: Arc, pretty: bool, } #[pymethods] impl PyUnparser { #[new] pub fn new(dialect: PyDialect) -> Self { Self { dialect: dialect.dialect.clone(), pretty: false, } } pub fn plan_to_sql(&self, plan: &PyLogicalPlan) -> PyResult { let mut unparser = Unparser::new(self.dialect.as_ref()); unparser = unparser.with_pretty(self.pretty); let sql = unparser .plan_to_sql(&plan.plan()) .map_err(|e| PyValueError::new_err(e.to_string()))?; Ok(sql.to_string()) } pub fn with_pretty(&self, pretty: bool) -> Self { Self { dialect: self.dialect.clone(), pretty, } } } pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> { m.add_class::()?; m.add_class::()?; Ok(()) } apache-datafusion-python-54.0.0/crates/util/000077500000000000000000000000001521727353300207415ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/util/Cargo.toml000066400000000000000000000024331521727353300226730ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. [package] name = "datafusion-python-util" version.workspace = true edition.workspace = true rust-version.workspace = true license.workspace = true description.workspace = true homepage.workspace = true repository.workspace = true [dependencies] tokio = { workspace = true, features = ["macros", "rt", "rt-multi-thread"] } pyo3 = { workspace = true } datafusion = { workspace = true } datafusion-ffi = { workspace = true } datafusion-proto = { workspace = true } arrow = { workspace = true } prost = { workspace = true } apache-datafusion-python-54.0.0/crates/util/src/000077500000000000000000000000001521727353300215305ustar00rootroot00000000000000apache-datafusion-python-54.0.0/crates/util/src/errors.rs000066400000000000000000000070561521727353300234220ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use core::fmt; use std::error::Error; use std::fmt::Debug; use datafusion::arrow::error::ArrowError; use datafusion::error::DataFusionError as InnerDataFusionError; use prost::EncodeError; use pyo3::PyErr; use pyo3::exceptions::{PyException, PyValueError}; pub type PyDataFusionResult = std::result::Result; #[derive(Debug)] pub enum PyDataFusionError { ExecutionError(Box), ArrowError(ArrowError), Common(String), PythonError(PyErr), EncodeError(EncodeError), } impl fmt::Display for PyDataFusionError { fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result { match self { PyDataFusionError::ExecutionError(e) => write!(f, "DataFusion error: {e}"), PyDataFusionError::ArrowError(e) => write!(f, "Arrow error: {e:?}"), PyDataFusionError::PythonError(e) => write!(f, "Python error {e:?}"), PyDataFusionError::Common(e) => write!(f, "{e}"), PyDataFusionError::EncodeError(e) => write!(f, "Failed to encode substrait plan: {e}"), } } } impl From for PyDataFusionError { fn from(err: ArrowError) -> PyDataFusionError { PyDataFusionError::ArrowError(err) } } impl From for PyDataFusionError { fn from(err: InnerDataFusionError) -> PyDataFusionError { PyDataFusionError::ExecutionError(Box::new(err)) } } impl From for PyDataFusionError { fn from(err: PyErr) -> PyDataFusionError { PyDataFusionError::PythonError(err) } } impl From for PyErr { fn from(err: PyDataFusionError) -> PyErr { match err { PyDataFusionError::PythonError(py_err) => py_err, _ => PyException::new_err(err.to_string()), } } } impl Error for PyDataFusionError {} pub fn py_type_err(e: impl Debug) -> PyErr { PyErr::new::(format!("{e:?}")) } pub fn py_runtime_err(e: impl Debug) -> PyErr { PyErr::new::(format!("{e:?}")) } pub fn py_datafusion_err(e: impl Debug) -> PyErr { PyErr::new::(format!("{e:?}")) } pub fn py_unsupported_variant_err(e: impl Debug) -> PyErr { PyErr::new::(format!("{e:?}")) } pub fn to_datafusion_err(e: impl Debug) -> InnerDataFusionError { InnerDataFusionError::Execution(format!("{e:?}")) } pub fn from_datafusion_error(err: InnerDataFusionError) -> PyErr { match err { InnerDataFusionError::External(boxed) => match boxed.downcast::() { Ok(py_err) => *py_err, Err(original_boxed) => PyValueError::new_err(format!("{original_boxed}")), }, _ => PyValueError::new_err(format!("{err}")), } } apache-datafusion-python-54.0.0/crates/util/src/lib.rs000066400000000000000000000323151521727353300226500ustar00rootroot00000000000000// Licensed to the Apache Software Foundation (ASF) under one // or more contributor license agreements. See the NOTICE file // distributed with this work for additional information // regarding copyright ownership. The ASF licenses this file // to you under the Apache License, Version 2.0 (the // "License"); you may not use this file except in compliance // with the License. You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, // software distributed under the License is distributed on an // "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY // KIND, either express or implied. See the License for the // specific language governing permissions and limitations // under the License. use std::future::Future; use std::ptr::NonNull; use std::sync::{Arc, OnceLock}; use std::time::Duration; use datafusion::datasource::TableProvider; use datafusion::execution::TaskContext; use datafusion::execution::context::SessionContext; use datafusion::logical_expr::Volatility; use datafusion::physical_optimizer::PhysicalOptimizerRule; use datafusion_ffi::execution::FFI_TaskContextProvider; use datafusion_ffi::physical_optimizer::FFI_PhysicalOptimizerRule; use datafusion_ffi::proto::logical_extension_codec::FFI_LogicalExtensionCodec; use datafusion_ffi::proto::physical_extension_codec::FFI_PhysicalExtensionCodec; use datafusion_ffi::table_provider::FFI_TableProvider; use datafusion_proto::physical_plan::PhysicalExtensionCodec; use pyo3::exceptions::{PyImportError, PyTypeError, PyValueError}; use pyo3::prelude::*; use pyo3::types::{PyCapsule, PyType}; use tokio::runtime::Runtime; use tokio::task::JoinHandle; use tokio::time::sleep; pub mod errors; pub use crate::errors::to_datafusion_err; use crate::errors::{PyDataFusionError, PyDataFusionResult}; /// Utility to get the Tokio Runtime from Python #[inline] pub fn get_tokio_runtime() -> &'static Runtime { // NOTE: Other pyo3 python libraries have had issues with using tokio // behind a forking app-server like `gunicorn` // If we run into that problem, in the future we can look to `delta-rs` // which adds a check in that disallows calls from a forked process // https://github.com/delta-io/delta-rs/blob/87010461cfe01563d91a4b9cd6fa468e2ad5f283/python/src/utils.rs#L10-L31 static RUNTIME: OnceLock = OnceLock::new(); RUNTIME.get_or_init(|| Runtime::new().unwrap()) } #[inline] pub fn is_ipython_env(py: Python) -> &'static bool { static IS_IPYTHON_ENV: OnceLock = OnceLock::new(); IS_IPYTHON_ENV.get_or_init(|| { py.import("IPython") .and_then(|ipython| ipython.call_method0("get_ipython")) .map(|ipython| !ipython.is_none()) .unwrap_or(false) }) } /// Utility to get the Global Datafussion CTX #[inline] pub fn get_global_ctx() -> &'static Arc { static CTX: OnceLock> = OnceLock::new(); CTX.get_or_init(|| Arc::new(SessionContext::new())) } /// Utility to collect rust futures with GIL released and respond to /// Python interrupts such as ``KeyboardInterrupt``. If a signal is /// received while the future is running, the future is aborted and the /// corresponding Python exception is raised. pub fn wait_for_future(py: Python, fut: F) -> PyResult where F: Future + Send, F::Output: Send, { let runtime: &Runtime = get_tokio_runtime(); const INTERVAL_CHECK_SIGNALS: Duration = Duration::from_millis(1_000); // Some fast running processes that generate many `wait_for_future` calls like // PartitionedDataFrameStreamReader::next require checking for interrupts early py.run(cr"pass", None, None)?; py.check_signals()?; py.detach(|| { runtime.block_on(async { tokio::pin!(fut); loop { tokio::select! { res = &mut fut => break Ok(res), _ = sleep(INTERVAL_CHECK_SIGNALS) => { Python::attach(|py| { // Execute a no-op Python statement to trigger signal processing. // This is necessary because py.check_signals() alone doesn't // actually check for signals - it only raises an exception if // a signal was already set during a previous Python API call. // Running even trivial Python code forces the interpreter to // process any pending signals (like KeyboardInterrupt). py.run(cr"pass", None, None)?; py.check_signals() })?; } } } }) }) } /// Spawn a [`Future`] on the Tokio runtime and wait for completion /// while respecting Python signal handling. pub fn spawn_future(py: Python, fut: F) -> PyDataFusionResult where F: Future> + Send + 'static, T: Send + 'static, { let rt = get_tokio_runtime(); let handle: JoinHandle> = rt.spawn(fut); // Wait for the join handle while respecting Python signal handling. // We handle errors in two steps so `?` maps the error types correctly: // 1) convert any Python-related error from `wait_for_future` into `PyDataFusionError` // 2) convert any DataFusion error (inner result) into `PyDataFusionError` let inner_result = wait_for_future(py, async { // handle.await yields `Result, JoinError>` // map JoinError into a DataFusion error so the async block returns // `datafusion::common::Result` (i.e. Result) match handle.await { Ok(inner) => inner, Err(join_err) => Err(to_datafusion_err(join_err)), } })?; // converts PyErr -> PyDataFusionError // `inner_result` is `datafusion::common::Result`; use `?` to convert // the inner DataFusion error into `PyDataFusionError` via `From` and // return the inner `T` on success. Ok(inner_result?) } pub fn parse_volatility(value: &str) -> PyDataFusionResult { Ok(match value { "immutable" => Volatility::Immutable, "stable" => Volatility::Stable, "volatile" => Volatility::Volatile, value => { return Err(PyDataFusionError::Common(format!( "Unsupported volatility type: `{value}`, supported \ values are: immutable, stable and volatile." ))); } }) } pub fn validate_pycapsule(capsule: &Bound, name: &str) -> PyResult<()> { let capsule_name = capsule.name()?; if capsule_name.is_none() { return Err(PyValueError::new_err(format!( "Expected {name} PyCapsule to have name set." ))); } let capsule_name = unsafe { capsule_name.unwrap().as_cstr().to_str()? }; if capsule_name != name { return Err(PyValueError::new_err(format!( "Expected name '{name}' in PyCapsule, instead got '{capsule_name}'" ))); } Ok(()) } pub fn table_provider_from_pycapsule<'py>( mut obj: Bound<'py, PyAny>, session: Bound<'py, PyAny>, ) -> PyResult>> { if obj.hasattr("__datafusion_table_provider__")? { obj = obj .getattr("__datafusion_table_provider__")? .call1((session,)).map_err(|err| { let py = obj.py(); if err.get_type(py).is(PyType::new::(py)) { PyImportError::new_err("Incompatible libraries. DataFusion 52.0.0 introduced an incompatible signature change for table providers. Either downgrade DataFusion or upgrade your function library.") } else { err } })?; } if let Ok(capsule) = obj.cast::() { let data: NonNull = capsule .pointer_checked(Some(c"datafusion_table_provider"))? .cast(); let provider = unsafe { data.as_ref() }; let provider: Arc = provider.into(); Ok(Some(provider)) } else { Ok(None) } } pub fn create_logical_extension_capsule<'py>( py: Python<'py>, codec: &FFI_LogicalExtensionCodec, ) -> PyResult> { let name = cr"datafusion_logical_extension_codec".into(); let codec = codec.clone(); PyCapsule::new(py, codec, Some(name)) } pub fn ffi_logical_codec_from_pycapsule(obj: Bound) -> PyResult { let attr_name = "__datafusion_logical_extension_codec__"; let capsule = if obj.hasattr(attr_name)? { obj.getattr(attr_name)?.call0()? } else { obj }; let capsule = capsule.cast::()?; let data: NonNull = capsule .pointer_checked(Some(c"datafusion_logical_extension_codec"))? .cast(); let codec = unsafe { data.as_ref() }; Ok(codec.clone()) } pub fn create_physical_extension_capsule<'py>( py: Python<'py>, codec: &FFI_PhysicalExtensionCodec, ) -> PyResult> { let name = cr"datafusion_physical_extension_codec".into(); let codec = codec.clone(); PyCapsule::new(py, codec, Some(name)) } /// Define a `(obj) -> PyResult>` extractor that /// accepts either a raw `PyCapsule` carrying `$ffi_type` or any object /// exposing `____()` that returns one. /// /// Use this when `Arc<$output_type>: From<&$ffi_type>` (infallible /// conversion). For fallible conversions use [`try_from_pycapsule!`] /// instead. #[macro_export] macro_rules! from_pycapsule { ($fn_name:ident, $capsule_name:literal, $ffi_type:ty, $output_type:ty) => { pub fn $fn_name( obj: &$crate::pyo3::Bound<$crate::pyo3::PyAny>, ) -> $crate::pyo3::PyResult> { use $crate::pyo3::prelude::*; use $crate::pyo3::types::PyCapsule; let mut obj = obj.clone(); if obj.hasattr(concat!("__", $capsule_name, "__"))? { obj = obj.getattr(concat!("__", $capsule_name, "__"))?.call0()?; } let capsule = obj.cast::().map_err(|_| { $crate::errors::py_datafusion_err(concat!( "Invalid ", $capsule_name, ". Does not contain PyCapsule object." )) })?; $crate::validate_pycapsule(&capsule, $capsule_name)?; let expected_name = std::ffi::CString::new($capsule_name) .expect("capsule name must not contain interior NUL bytes"); let data: std::ptr::NonNull<$ffi_type> = capsule .pointer_checked(Some(expected_name.as_c_str()))? .cast(); let output_obj = unsafe { data.as_ref() }; let output_obj: std::sync::Arc<$output_type> = output_obj.into(); Ok(output_obj) } }; } /// Same shape as [`from_pycapsule!`] but for FFI types whose conversion /// into `Arc<$output_type>` is fallible (uses `TryFrom`). #[macro_export] macro_rules! try_from_pycapsule { ($fn_name:ident, $capsule_name:literal, $ffi_type:ty, $output_type:ty) => { pub fn $fn_name( obj: &$crate::pyo3::Bound<$crate::pyo3::PyAny>, ) -> $crate::pyo3::PyResult> { use $crate::pyo3::prelude::*; use $crate::pyo3::types::PyCapsule; let mut obj = obj.clone(); if obj.hasattr(concat!("__", $capsule_name, "__"))? { obj = obj.getattr(concat!("__", $capsule_name, "__"))?.call0()?; } let capsule = obj.cast::().map_err(|_| { $crate::errors::py_datafusion_err(concat!( "Invalid ", $capsule_name, ". Does not contain PyCapsule object." )) })?; $crate::validate_pycapsule(&capsule, $capsule_name)?; let expected_name = std::ffi::CString::new($capsule_name) .expect("capsule name must not contain interior NUL bytes"); let data: std::ptr::NonNull<$ffi_type> = capsule .pointer_checked(Some(expected_name.as_c_str()))? .cast(); let output_obj = unsafe { data.as_ref() }; let output_obj: std::sync::Arc<$output_type> = output_obj .try_into() .map_err($crate::errors::py_datafusion_err)?; Ok(output_obj) } }; } // Re-export pyo3 so the macros expand inside downstream crates without // requiring an explicit pyo3 dep at the call site. #[doc(hidden)] pub use pyo3; from_pycapsule!( physical_codec_from_pycapsule, "datafusion_physical_extension_codec", FFI_PhysicalExtensionCodec, dyn PhysicalExtensionCodec ); from_pycapsule!( physical_optimizer_rule_from_pycapsule, "datafusion_physical_optimizer_rule", FFI_PhysicalOptimizerRule, dyn PhysicalOptimizerRule + Send + Sync ); try_from_pycapsule!( task_context_from_pycapsule, "datafusion_task_context_provider", FFI_TaskContextProvider, TaskContext ); apache-datafusion-python-54.0.0/dev/000077500000000000000000000000001521727353300172615ustar00rootroot00000000000000apache-datafusion-python-54.0.0/dev/build-set-env.sh000077500000000000000000000015701521727353300223010ustar00rootroot00000000000000#!/bin/bash # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. export PY_DATAFUSION_VERSION=$(awk -F'[ ="]+' '$1 == "version" { print $2 }' Cargo.toml) apache-datafusion-python-54.0.0/dev/changelog/000077500000000000000000000000001521727353300212105ustar00rootroot00000000000000apache-datafusion-python-54.0.0/dev/changelog/43.0.0.md000066400000000000000000000105241521727353300222560ustar00rootroot00000000000000 # Apache DataFusion Python 43.0.0 Changelog This release consists of 26 commits from 7 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: expose `drop` method [#913](https://github.com/apache/datafusion-python/pull/913) (ion-elgreco) - feat: expose `join_on` [#914](https://github.com/apache/datafusion-python/pull/914) (ion-elgreco) - feat: add fill_null/nan expressions [#919](https://github.com/apache/datafusion-python/pull/919) (ion-elgreco) - feat: add `with_columns` [#909](https://github.com/apache/datafusion-python/pull/909) (ion-elgreco) - feat: add `cast` to DataFrame [#916](https://github.com/apache/datafusion-python/pull/916) (ion-elgreco) - feat: add `head`, `tail` methods [#915](https://github.com/apache/datafusion-python/pull/915) (ion-elgreco) **Fixed bugs:** - fix: remove use of deprecated `make_scalar_function` [#906](https://github.com/apache/datafusion-python/pull/906) (Michael-J-Ward) - fix: udwf example [#948](https://github.com/apache/datafusion-python/pull/948) (mesejo) **Other:** - Ts/minor updates release process [#903](https://github.com/apache/datafusion-python/pull/903) (timsaucer) - build(deps): bump pyo3 from 0.22.3 to 0.22.4 [#910](https://github.com/apache/datafusion-python/pull/910) (dependabot[bot]) - refactor: `from_arrow` use protocol typehints [#917](https://github.com/apache/datafusion-python/pull/917) (ion-elgreco) - Change requires-python version in pyproject.toml [#924](https://github.com/apache/datafusion-python/pull/924) (kosiew) - chore: deprecate `select_columns` [#911](https://github.com/apache/datafusion-python/pull/911) (ion-elgreco) - build(deps): bump uuid from 1.10.0 to 1.11.0 [#927](https://github.com/apache/datafusion-python/pull/927) (dependabot[bot]) - Add array_empty scalar function [#931](https://github.com/apache/datafusion-python/pull/931) (kosiew) - add `cardinality` function to calculate total distinct elements in an array [#937](https://github.com/apache/datafusion-python/pull/937) (kosiew) - Add empty scalar function (alias of array_empty), fix a small typo [#938](https://github.com/apache/datafusion-python/pull/938) (kosiew) - README How to develop section now also works on Apple M1 [#940](https://github.com/apache/datafusion-python/pull/940) (drauschenbach) - refactor: dataframe `join` params [#912](https://github.com/apache/datafusion-python/pull/912) (ion-elgreco) - Upgrade to Datafusion 43 [#905](https://github.com/apache/datafusion-python/pull/905) (Michael-J-Ward) - build(deps): bump tokio from 1.40.0 to 1.41.1 [#946](https://github.com/apache/datafusion-python/pull/946) (dependabot[bot]) - Add list_cat, list_concat, list_repeat [#942](https://github.com/apache/datafusion-python/pull/942) (kosiew) - Add foreign table providers [#921](https://github.com/apache/datafusion-python/pull/921) (timsaucer) - Add make_list and tests for make_list, make_array [#949](https://github.com/apache/datafusion-python/pull/949) (kosiew) - Documentation updates: simplify examples and add section on data sources [#955](https://github.com/apache/datafusion-python/pull/955) (timsaucer) - Add datafusion.extract [#959](https://github.com/apache/datafusion-python/pull/959) (kosiew) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 9 Ion Koutsouris 7 kosiew 3 Tim Saucer 3 dependabot[bot] 2 Michael J Ward 1 Daniel Mesejo 1 David Rauschenbach ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/44.0.0.md000066400000000000000000000052461521727353300222640ustar00rootroot00000000000000 # Apache DataFusion Python 44.0.0 Changelog This release consists of 12 commits from 5 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: support enable_url_table config [#980](https://github.com/apache/datafusion-python/pull/980) (chenkovsky) - feat: remove DataFusion pyarrow feat [#1000](https://github.com/apache/datafusion-python/pull/1000) (timsaucer) **Fixed bugs:** - fix: correct LZ0 to LZO in compression options [#995](https://github.com/apache/datafusion-python/pull/995) (kosiew) **Other:** - Add arrow cast [#962](https://github.com/apache/datafusion-python/pull/962) (kosiew) - Fix small issues in pyproject.toml [#976](https://github.com/apache/datafusion-python/pull/976) (kylebarron) - chore: set validation and type hint for ffi tableprovider [#983](https://github.com/apache/datafusion-python/pull/983) (ion-elgreco) - Support async iteration of RecordBatchStream [#975](https://github.com/apache/datafusion-python/pull/975) (kylebarron) - Chore/upgrade datafusion 44 [#973](https://github.com/apache/datafusion-python/pull/973) (timsaucer) - Default to ZSTD compression when writing Parquet [#981](https://github.com/apache/datafusion-python/pull/981) (kosiew) - Feat/use uv python management [#994](https://github.com/apache/datafusion-python/pull/994) (timsaucer) - minor: Update dependencies prior to release [#999](https://github.com/apache/datafusion-python/pull/999) (timsaucer) - Apply import ordering in ruff check [#1001](https://github.com/apache/datafusion-python/pull/1001) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 5 Tim Saucer 3 kosiew 2 Kyle Barron 1 Chongchen Chen 1 Ion Koutsouris ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/45.0.0.md000066400000000000000000000027121521727353300222600ustar00rootroot00000000000000 # Apache DataFusion Python 45.0.0 Changelog This release consists of 2 commits from 2 contributors. See credits at the end of this changelog for more information. **Fixed bugs:** - fix: add to_timestamp_nanos [#1020](https://github.com/apache/datafusion-python/pull/1020) (chenkovsky) **Other:** - Chore/upgrade datafusion 45 [#1010](https://github.com/apache/datafusion-python/pull/1010) (kevinjqliu) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 1 Kevin Liu 1 Tim Saucer ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/46.0.0.md000066400000000000000000000074671521727353300222750ustar00rootroot00000000000000 # Apache DataFusion Python 46.0.0 Changelog This release consists of 21 commits from 11 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: reads using global ctx [#982](https://github.com/apache/datafusion-python/pull/982) (ion-elgreco) - feat: Implementation of udf and udaf decorator [#1040](https://github.com/apache/datafusion-python/pull/1040) (CrystalZhou0529) - feat: expose regex_count function [#1066](https://github.com/apache/datafusion-python/pull/1066) (nirnayroy) - feat: Update DataFusion dependency to 46 [#1079](https://github.com/apache/datafusion-python/pull/1079) (timsaucer) **Fixed bugs:** - fix: add to_timestamp_nanos [#1020](https://github.com/apache/datafusion-python/pull/1020) (chenkovsky) - fix: type checking [#993](https://github.com/apache/datafusion-python/pull/993) (chenkovsky) **Other:** - [infra] Fail Clippy on rust build warnings [#1029](https://github.com/apache/datafusion-python/pull/1029) (kevinjqliu) - Add user documentation for the FFI approach [#1031](https://github.com/apache/datafusion-python/pull/1031) (timsaucer) - build(deps): bump arrow from 54.1.0 to 54.2.0 [#1035](https://github.com/apache/datafusion-python/pull/1035) (dependabot[bot]) - Chore: Release datafusion-python 45 [#1024](https://github.com/apache/datafusion-python/pull/1024) (timsaucer) - Enable Dataframe to be converted into views which can be used in register_table [#1016](https://github.com/apache/datafusion-python/pull/1016) (kosiew) - Add ruff check for missing futures import [#1052](https://github.com/apache/datafusion-python/pull/1052) (timsaucer) - Enable take comments to assign issues to users [#1058](https://github.com/apache/datafusion-python/pull/1058) (timsaucer) - Update python min version to 3.9 [#1043](https://github.com/apache/datafusion-python/pull/1043) (kevinjqliu) - feat/improve ruff test coverage [#1055](https://github.com/apache/datafusion-python/pull/1055) (timsaucer) - feat/making global context accessible for users [#1060](https://github.com/apache/datafusion-python/pull/1060) (jsai28) - Renaming Internal Structs [#1059](https://github.com/apache/datafusion-python/pull/1059) (Spaarsh) - test: add pytest asyncio tests [#1063](https://github.com/apache/datafusion-python/pull/1063) (jsai28) - Add decorator for udwf [#1061](https://github.com/apache/datafusion-python/pull/1061) (kosiew) - Add additional ruff suggestions [#1062](https://github.com/apache/datafusion-python/pull/1062) (Spaarsh) - Improve collection during repr and repr_html [#1036](https://github.com/apache/datafusion-python/pull/1036) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 7 Tim Saucer 2 Kevin Liu 2 Spaarsh 2 jsai28 2 kosiew 1 Chen Chongchen 1 Chongchen Chen 1 Crystal Zhou 1 Ion Koutsouris 1 Nirnay Roy 1 dependabot[bot] ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/47.0.0.md000066400000000000000000000067541521727353300222740ustar00rootroot00000000000000 # Apache DataFusion Python 47.0.0 Changelog This release consists of 23 commits from 5 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: support unparser [#1088](https://github.com/apache/datafusion-python/pull/1088) (chenkovsky) - feat: update datafusion dependency 47 [#1107](https://github.com/apache/datafusion-python/pull/1107) (timsaucer) - feat: alias with metadata [#1111](https://github.com/apache/datafusion-python/pull/1111) (chenkovsky) - feat: add missing PyLogicalPlan to_variant [#1085](https://github.com/apache/datafusion-python/pull/1085) (chenkovsky) - feat: add user defined table function support [#1113](https://github.com/apache/datafusion-python/pull/1113) (timsaucer) **Fixed bugs:** - fix: recursive import [#1117](https://github.com/apache/datafusion-python/pull/1117) (chenkovsky) **Other:** - Update changelog and version number [#1089](https://github.com/apache/datafusion-python/pull/1089) (timsaucer) - Documentation updates: mention correct dataset on basics page [#1081](https://github.com/apache/datafusion-python/pull/1081) (floscha) - Add Configurable HTML Table Formatter for DataFusion DataFrames in Python [#1100](https://github.com/apache/datafusion-python/pull/1100) (kosiew) - Add DataFrame usage guide with HTML rendering customization options [#1108](https://github.com/apache/datafusion-python/pull/1108) (kosiew) - 1075/enhancement/Make col class with __getattr__ [#1076](https://github.com/apache/datafusion-python/pull/1076) (deanm0000) - 1064/enhancement/add functions to Expr class [#1074](https://github.com/apache/datafusion-python/pull/1074) (deanm0000) - ci: require approving review [#1122](https://github.com/apache/datafusion-python/pull/1122) (timsaucer) - Partial fix for 1078: Enhance DataFrame Formatter Configuration with Memory and Display Controls [#1119](https://github.com/apache/datafusion-python/pull/1119) (kosiew) - Add fill_null method to DataFrame API for handling missing values [#1019](https://github.com/apache/datafusion-python/pull/1019) (kosiew) - minor: reduce error size [#1126](https://github.com/apache/datafusion-python/pull/1126) (timsaucer) - Move the udf module to user_defined [#1112](https://github.com/apache/datafusion-python/pull/1112) (timsaucer) - add unit tests for expression functions [#1121](https://github.com/apache/datafusion-python/pull/1121) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 12 Tim Saucer 4 Chen Chongchen 4 kosiew 2 deanm0000 1 Florian Schäfer ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/48.0.0.md000066400000000000000000000063301521727353300222630ustar00rootroot00000000000000 # Apache DataFusion Python 48.0.0 Changelog This release consists of 15 commits from 6 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: upgrade df48 dependency [#1143](https://github.com/apache/datafusion-python/pull/1143) (timsaucer) - feat: Support Parquet writer options [#1123](https://github.com/apache/datafusion-python/pull/1123) (nuno-faria) - feat: dataframe string formatter [#1170](https://github.com/apache/datafusion-python/pull/1170) (timsaucer) - feat: collect once during display() in jupyter notebooks [#1167](https://github.com/apache/datafusion-python/pull/1167) (timsaucer) - feat: python based catalog and schema provider [#1156](https://github.com/apache/datafusion-python/pull/1156) (timsaucer) - feat: add FFI support for user defined functions [#1145](https://github.com/apache/datafusion-python/pull/1145) (timsaucer) **Other:** - Release DataFusion 47.0.0 [#1130](https://github.com/apache/datafusion-python/pull/1130) (timsaucer) - Add a documentation build step in CI [#1139](https://github.com/apache/datafusion-python/pull/1139) (crystalxyz) - Add DataFrame API Documentation for DataFusion Python [#1132](https://github.com/apache/datafusion-python/pull/1132) (kosiew) - Add Interruptible Query Execution in Jupyter via KeyboardInterrupt Support [#1141](https://github.com/apache/datafusion-python/pull/1141) (kosiew) - Support types other than String and Int for partition columns [#1154](https://github.com/apache/datafusion-python/pull/1154) (miclegr) - Fix signature of `__arrow_c_stream__` [#1168](https://github.com/apache/datafusion-python/pull/1168) (kylebarron) - Consolidate DataFrame Docs: Merge HTML Rendering Section as Subpage [#1161](https://github.com/apache/datafusion-python/pull/1161) (kosiew) - Add compression_level support to ParquetWriterOptions and enhance write_parquet to accept full options object [#1169](https://github.com/apache/datafusion-python/pull/1169) (kosiew) - Simplify HTML Formatter Style Handling Using Script Injection [#1177](https://github.com/apache/datafusion-python/pull/1177) (kosiew) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 6 Tim Saucer 5 kosiew 1 Crystal Zhou 1 Kyle Barron 1 Michele Gregori 1 Nuno Faria ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/49.0.0.md000066400000000000000000000064611521727353300222710ustar00rootroot00000000000000 # Apache DataFusion Python 49.0.0 Changelog This release consists of 16 commits from 7 contributors. See credits at the end of this changelog for more information. **Fixed bugs:** - fix(build): Include build.rs in published crates [#1199](https://github.com/apache/datafusion-python/pull/1199) (colinmarc) **Other:** - 48.0.0 Release [#1175](https://github.com/apache/datafusion-python/pull/1175) (timsaucer) - Update CI rules [#1188](https://github.com/apache/datafusion-python/pull/1188) (timsaucer) - Fix Python UDAF Accumulator Interface example to Properly Handle State and Updates with List[Array] Types [#1192](https://github.com/apache/datafusion-python/pull/1192) (kosiew) - chore: Upgrade datafusion to version 49 [#1200](https://github.com/apache/datafusion-python/pull/1200) (nuno-faria) - Update how to dev instructions [#1179](https://github.com/apache/datafusion-python/pull/1179) (ntjohnson1) - build(deps): bump object_store from 0.12.2 to 0.12.3 [#1189](https://github.com/apache/datafusion-python/pull/1189) (dependabot[bot]) - build(deps): bump uuid from 1.17.0 to 1.18.0 [#1202](https://github.com/apache/datafusion-python/pull/1202) (dependabot[bot]) - build(deps): bump async-trait from 0.1.88 to 0.1.89 [#1203](https://github.com/apache/datafusion-python/pull/1203) (dependabot[bot]) - build(deps): bump slab from 0.4.10 to 0.4.11 [#1205](https://github.com/apache/datafusion-python/pull/1205) (dependabot[bot]) - Improved window and aggregate function signature [#1187](https://github.com/apache/datafusion-python/pull/1187) (timsaucer) - Optional improvements in verification instructions [#1183](https://github.com/apache/datafusion-python/pull/1183) (paleolimbot) - Improve `show()` output for empty DataFrames [#1208](https://github.com/apache/datafusion-python/pull/1208) (kosiew) - build(deps): bump actions/download-artifact from 4 to 5 [#1201](https://github.com/apache/datafusion-python/pull/1201) (dependabot[bot]) - build(deps): bump url from 2.5.4 to 2.5.7 [#1210](https://github.com/apache/datafusion-python/pull/1210) (dependabot[bot]) - build(deps): bump actions/checkout from 4 to 5 [#1204](https://github.com/apache/datafusion-python/pull/1204) (dependabot[bot]) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 7 dependabot[bot] 3 Tim Saucer 2 kosiew 1 Colin Marc 1 Dewey Dunnington 1 Nick 1 Nuno Faria ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/50.0.0.md000066400000000000000000000055231521727353300222570ustar00rootroot00000000000000 # Apache DataFusion Python 50.0.0 Changelog This release consists of 12 commits from 7 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: allow passing a slice to and expression with the [] indexing [#1215](https://github.com/apache/datafusion-python/pull/1215) (timsaucer) **Documentation updates:** - docs: fix CaseBuilder documentation example [#1225](https://github.com/apache/datafusion-python/pull/1225) (IndexSeek) - docs: update link to user example for custom table provider [#1224](https://github.com/apache/datafusion-python/pull/1224) (IndexSeek) - docs: add apache iceberg as datafusion data source [#1240](https://github.com/apache/datafusion-python/pull/1240) (kevinjqliu) **Other:** - 49.0.0 release [#1211](https://github.com/apache/datafusion-python/pull/1211) (timsaucer) - Update development guide in README.md [#1213](https://github.com/apache/datafusion-python/pull/1213) (YKoustubhRao) - Add benchmark script and documentation for maximizing CPU usage in DataFusion Python [#1216](https://github.com/apache/datafusion-python/pull/1216) (kosiew) - Fixing a few Typos [#1220](https://github.com/apache/datafusion-python/pull/1220) (ntjohnson1) - Set fail on warning for documentation generation [#1218](https://github.com/apache/datafusion-python/pull/1218) (timsaucer) - chore: remove redundant error transformation [#1232](https://github.com/apache/datafusion-python/pull/1232) (mesejo) - Support string column identifiers for sort/aggregate/window and stricter Expr validation [#1221](https://github.com/apache/datafusion-python/pull/1221) (kosiew) - Prepare for DF50 [#1231](https://github.com/apache/datafusion-python/pull/1231) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 4 Tim Saucer 2 Tyler White 2 kosiew 1 Daniel Mesejo 1 Kevin Liu 1 Koustubh Rao 1 Nick ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/50.1.0.md000066400000000000000000000047141521727353300222610ustar00rootroot00000000000000 # Apache DataFusion Python 50.1.0 Changelog This release consists of 11 commits from 7 contributors. See credits at the end of this changelog for more information. **Breaking changes:** - Unify Table representations [#1256](https://github.com/apache/datafusion-python/pull/1256) (timsaucer) **Implemented enhancements:** - feat: expose DataFrame.write_table [#1264](https://github.com/apache/datafusion-python/pull/1264) (timsaucer) - feat: expose` DataFrame.parse_sql_expr` [#1274](https://github.com/apache/datafusion-python/pull/1274) (milenkovicm) **Other:** - Update version number, add changelog [#1249](https://github.com/apache/datafusion-python/pull/1249) (timsaucer) - Fix drop() method to handle quoted column names consistently [#1242](https://github.com/apache/datafusion-python/pull/1242) (H0TB0X420) - Make Session Context `pyclass` frozen so interior mutability is only managed by rust [#1248](https://github.com/apache/datafusion-python/pull/1248) (ntjohnson1) - macos-13 is deprecated [#1259](https://github.com/apache/datafusion-python/pull/1259) (kevinjqliu) - Freeze PyO3 wrappers & introduce interior mutability to avoid PyO3 borrow errors [#1253](https://github.com/apache/datafusion-python/pull/1253) (kosiew) - chore: update dependencies [#1269](https://github.com/apache/datafusion-python/pull/1269) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 4 Tim Saucer 2 Siew Kam Onn 1 H0TB0X420 1 Kevin Liu 1 Marko Milenković 1 Nick 1 kosiew ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/51.0.0.md000066400000000000000000000102741521727353300222570ustar00rootroot00000000000000 # Apache DataFusion Python 51.0.0 Changelog This release consists of 23 commits from 7 contributors. See credits at the end of this changelog for more information. **Breaking changes:** - feat: reduce duplicate fields on join [#1184](https://github.com/apache/datafusion-python/pull/1184) (timsaucer) **Implemented enhancements:** - feat: expose `select_exprs` method on DataFrame [#1271](https://github.com/apache/datafusion-python/pull/1271) (milenkovicm) - feat: allow DataFrame.filter to accept SQL strings [#1276](https://github.com/apache/datafusion-python/pull/1276) (K-dash) - feat: add temporary view option for into_view [#1267](https://github.com/apache/datafusion-python/pull/1267) (timsaucer) - feat: support session token parameter for AmazonS3 [#1275](https://github.com/apache/datafusion-python/pull/1275) (GCHQDeveloper028) - feat: `with_column` supports SQL expression [#1284](https://github.com/apache/datafusion-python/pull/1284) (milenkovicm) - feat: Add SQL expression for `repartition_by_hash` [#1285](https://github.com/apache/datafusion-python/pull/1285) (milenkovicm) - feat: Add SQL expression support for `with_columns` [#1286](https://github.com/apache/datafusion-python/pull/1286) (milenkovicm) **Fixed bugs:** - fix: use coalesce instead of drop_duplicate_keys for join [#1318](https://github.com/apache/datafusion-python/pull/1318) (mesejo) - fix: Inconsistent schemas when converting to pyarrow [#1315](https://github.com/apache/datafusion-python/pull/1315) (nuno-faria) **Other:** - Release 50.1 [#1281](https://github.com/apache/datafusion-python/pull/1281) (timsaucer) - Update python minimum version to 3.10 [#1296](https://github.com/apache/datafusion-python/pull/1296) (timsaucer) - chore: update datafusion minor version [#1297](https://github.com/apache/datafusion-python/pull/1297) (timsaucer) - Enable remaining pylints [#1298](https://github.com/apache/datafusion-python/pull/1298) (timsaucer) - Add Arrow C streaming, DataFrame iteration, and OOM-safe streaming execution [#1222](https://github.com/apache/datafusion-python/pull/1222) (kosiew) - Add PyCapsule Type Support and Type Hint Enhancements for AggregateUDF in DataFusion Python Bindings [#1277](https://github.com/apache/datafusion-python/pull/1277) (kosiew) - Add collect_column to dataframe [#1302](https://github.com/apache/datafusion-python/pull/1302) (timsaucer) - chore: apply cargo fmt with import organization [#1303](https://github.com/apache/datafusion-python/pull/1303) (timsaucer) - Feat/parameterized sql queries [#964](https://github.com/apache/datafusion-python/pull/964) (timsaucer) - Upgrade to Datafusion 51 [#1311](https://github.com/apache/datafusion-python/pull/1311) (nuno-faria) - minor: resolve build errors after latest merge into main [#1325](https://github.com/apache/datafusion-python/pull/1325) (timsaucer) - Update build workflow link [#1330](https://github.com/apache/datafusion-python/pull/1330) (timsaucer) - Do not convert pyarrow scalar values to plain python types when passing as `lit` [#1319](https://github.com/apache/datafusion-python/pull/1319) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 12 Tim Saucer 4 Marko Milenković 2 Nuno Faria 2 kosiew 1 Daniel Mesejo 1 GCHQDeveloper028 1 𝕂 ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/52.0.0.md000066400000000000000000000110511521727353300222520ustar00rootroot00000000000000 # Apache DataFusion Python 52.0.0 Changelog This release consists of 26 commits from 9 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: add CatalogProviderList support [#1363](https://github.com/apache/datafusion-python/pull/1363) (timsaucer) - feat: add support for generating JSON formatted substrait plan [#1376](https://github.com/apache/datafusion-python/pull/1376) (Prathamesh9284) - feat: add regexp_instr function [#1382](https://github.com/apache/datafusion-python/pull/1382) (mesejo) **Fixed bugs:** - fix: mangled errors [#1377](https://github.com/apache/datafusion-python/pull/1377) (mesejo) **Documentation updates:** - docs: Clarify first_value usage in select vs aggregate [#1348](https://github.com/apache/datafusion-python/pull/1348) (AdMub) **Other:** - Release 51.0.0 [#1333](https://github.com/apache/datafusion-python/pull/1333) (timsaucer) - Use explicit timer in unit test [#1338](https://github.com/apache/datafusion-python/pull/1338) (timsaucer) - Add use_fabric_endpoint parameter to MicrosoftAzure class [#1357](https://github.com/apache/datafusion-python/pull/1357) (djouallah) - Prepare for DF52 release [#1337](https://github.com/apache/datafusion-python/pull/1337) (timsaucer) - build(deps): bump actions/checkout from 5 to 6 [#1310](https://github.com/apache/datafusion-python/pull/1310) (dependabot[bot]) - build(deps): bump actions/download-artifact from 5 to 7 [#1321](https://github.com/apache/datafusion-python/pull/1321) (dependabot[bot]) - build(deps): bump actions/upload-artifact from 4 to 6 [#1322](https://github.com/apache/datafusion-python/pull/1322) (dependabot[bot]) - build(deps): bump actions/cache from 4 to 5 [#1323](https://github.com/apache/datafusion-python/pull/1323) (dependabot[bot]) - Pass Field information back and forth when using scalar UDFs [#1299](https://github.com/apache/datafusion-python/pull/1299) (timsaucer) - Update dependency minor versions to prepare for DF52 release [#1368](https://github.com/apache/datafusion-python/pull/1368) (timsaucer) - Improve displayed error by using `DataFusionError`'s `Display` trait [#1370](https://github.com/apache/datafusion-python/pull/1370) (abey79) - Enforce DataFrame display memory limits with `max_rows` + `min_rows` constraint (deprecate `repr_rows`) [#1367](https://github.com/apache/datafusion-python/pull/1367) (kosiew) - Implement all CSV reader options [#1361](https://github.com/apache/datafusion-python/pull/1361) (timsaucer) - chore: add confirmation before tarball is released [#1372](https://github.com/apache/datafusion-python/pull/1372) (milenkovicm) - Build in debug mode for PRs [#1375](https://github.com/apache/datafusion-python/pull/1375) (timsaucer) - minor: remove ffi test wheel from distribution artifact [#1378](https://github.com/apache/datafusion-python/pull/1378) (timsaucer) - chore: update rust 2024 edition [#1371](https://github.com/apache/datafusion-python/pull/1371) (timsaucer) - Fix Python UDAF list-of-timestamps return by enforcing list-valued scalars and caching PyArrow types [#1347](https://github.com/apache/datafusion-python/pull/1347) (kosiew) - minor: update cargo dependencies [#1383](https://github.com/apache/datafusion-python/pull/1383) (timsaucer) - chore: bump Python version for RAT checking [#1386](https://github.com/apache/datafusion-python/pull/1386) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 13 Tim Saucer 4 dependabot[bot] 2 Daniel Mesejo 2 kosiew 1 Adisa Mubarak (AdMub) 1 Antoine Beyeler 1 Dhanashri Prathamesh Iranna 1 Marko Milenković 1 Mimoune ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/53.0.0.md000066400000000000000000000172631521727353300222660ustar00rootroot00000000000000 # Apache DataFusion Python 53.0.0 Changelog This release consists of 52 commits from 9 contributors. See credits at the end of this changelog for more information. **Breaking changes:** - minor: remove deprecated interfaces [#1481](https://github.com/apache/datafusion-python/pull/1481) (timsaucer) **Implemented enhancements:** - feat: feat: add to_time, to_local_time, to_date functions [#1387](https://github.com/apache/datafusion-python/pull/1387) (mesejo) - feat: Add FFI_TableProviderFactory support [#1396](https://github.com/apache/datafusion-python/pull/1396) (davisp) **Fixed bugs:** - fix: satisfy rustfmt check in lib.rs re-exports [#1406](https://github.com/apache/datafusion-python/pull/1406) (kevinjqliu) **Documentation updates:** - docs: clarify DataFusion 52 FFI session-parameter requirement for provider hooks [#1439](https://github.com/apache/datafusion-python/pull/1439) (kevinjqliu) **Other:** - Merge release 52.0.0 into main [#1389](https://github.com/apache/datafusion-python/pull/1389) (timsaucer) - Add workflow to verify release candidate on multiple systems [#1388](https://github.com/apache/datafusion-python/pull/1388) (timsaucer) - Allow running "verify release candidate" github workflow on Windows [#1392](https://github.com/apache/datafusion-python/pull/1392) (kevinjqliu) - ci: update pre-commit hooks, fix linting, and refresh dependencies [#1385](https://github.com/apache/datafusion-python/pull/1385) (dariocurr) - Add CI check for crates.io patches [#1407](https://github.com/apache/datafusion-python/pull/1407) (timsaucer) - Enable doc tests in local and CI testing [#1409](https://github.com/apache/datafusion-python/pull/1409) (ntjohnson1) - Upgrade to DataFusion 53 [#1402](https://github.com/apache/datafusion-python/pull/1402) (nuno-faria) - Catch warnings in FFI unit tests [#1410](https://github.com/apache/datafusion-python/pull/1410) (timsaucer) - Add docstring examples for Scalar trigonometric functions [#1411](https://github.com/apache/datafusion-python/pull/1411) (ntjohnson1) - Create workspace with core and util crates [#1414](https://github.com/apache/datafusion-python/pull/1414) (timsaucer) - Add docstring examples for Scalar regex, crypto, struct and other [#1422](https://github.com/apache/datafusion-python/pull/1422) (ntjohnson1) - Add docstring examples for Scalar math functions [#1421](https://github.com/apache/datafusion-python/pull/1421) (ntjohnson1) - Add docstring examples for Common utility functions [#1419](https://github.com/apache/datafusion-python/pull/1419) (ntjohnson1) - Add docstring examples for Aggregate basic and bitwise/boolean functions [#1416](https://github.com/apache/datafusion-python/pull/1416) (ntjohnson1) - Fix CI errors on main [#1432](https://github.com/apache/datafusion-python/pull/1432) (timsaucer) - Add docstring examples for Scalar temporal functions [#1424](https://github.com/apache/datafusion-python/pull/1424) (ntjohnson1) - Add docstring examples for Aggregate statistical and regression functions [#1417](https://github.com/apache/datafusion-python/pull/1417) (ntjohnson1) - Add docstring examples for Scalar array/list functions [#1420](https://github.com/apache/datafusion-python/pull/1420) (ntjohnson1) - Add docstring examples for Scalar string functions [#1423](https://github.com/apache/datafusion-python/pull/1423) (ntjohnson1) - Add docstring examples for Aggregate window functions [#1418](https://github.com/apache/datafusion-python/pull/1418) (ntjohnson1) - ci: pin third-party actions to Apache-approved SHAs [#1438](https://github.com/apache/datafusion-python/pull/1438) (kevinjqliu) - minor: bump datafusion to release version [#1441](https://github.com/apache/datafusion-python/pull/1441) (timsaucer) - ci: add swap during build, use tpchgen-cli [#1443](https://github.com/apache/datafusion-python/pull/1443) (timsaucer) - Update remaining existing examples to make testable/standalone executable [#1437](https://github.com/apache/datafusion-python/pull/1437) (ntjohnson1) - Do not run validate_pycapsule if pointer_checked is used [#1426](https://github.com/apache/datafusion-python/pull/1426) (Tpt) - Implement configuration extension support [#1391](https://github.com/apache/datafusion-python/pull/1391) (timsaucer) - Add a working, more complete example of using a catalog (docs) [#1427](https://github.com/apache/datafusion-python/pull/1427) (toppyy) - chore: update dependencies [#1447](https://github.com/apache/datafusion-python/pull/1447) (timsaucer) - Complete doc string examples for functions.py [#1435](https://github.com/apache/datafusion-python/pull/1435) (ntjohnson1) - chore: enforce uv lockfile consistency in CI and pre-commit [#1398](https://github.com/apache/datafusion-python/pull/1398) (mesejo) - CI: Add CodeQL workflow for GitHub Actions security scanning [#1408](https://github.com/apache/datafusion-python/pull/1408) (kevinjqliu) - ci: update codespell paths [#1469](https://github.com/apache/datafusion-python/pull/1469) (timsaucer) - Add missing datetime functions [#1467](https://github.com/apache/datafusion-python/pull/1467) (timsaucer) - Add AI skill to check current repository against upstream APIs [#1460](https://github.com/apache/datafusion-python/pull/1460) (timsaucer) - Add missing string function `contains` [#1465](https://github.com/apache/datafusion-python/pull/1465) (timsaucer) - Add missing conditional functions [#1464](https://github.com/apache/datafusion-python/pull/1464) (timsaucer) - Reduce peak memory usage during release builds to fix OOM on manylinux runners [#1445](https://github.com/apache/datafusion-python/pull/1445) (kevinjqliu) - Add missing map functions [#1461](https://github.com/apache/datafusion-python/pull/1461) (timsaucer) - minor: Fix pytest instructions in the README [#1477](https://github.com/apache/datafusion-python/pull/1477) (nuno-faria) - Add missing array functions [#1468](https://github.com/apache/datafusion-python/pull/1468) (timsaucer) - Add missing scalar functions [#1470](https://github.com/apache/datafusion-python/pull/1470) (timsaucer) - Add missing aggregate functions [#1471](https://github.com/apache/datafusion-python/pull/1471) (timsaucer) - Add missing Dataframe functions [#1472](https://github.com/apache/datafusion-python/pull/1472) (timsaucer) - Add missing deregister methods to SessionContext [#1473](https://github.com/apache/datafusion-python/pull/1473) (timsaucer) - Add missing registration methods [#1474](https://github.com/apache/datafusion-python/pull/1474) (timsaucer) - Add missing SessionContext utility methods [#1475](https://github.com/apache/datafusion-python/pull/1475) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 25 Tim Saucer 13 Nick 6 Kevin Liu 2 Daniel Mesejo 2 Nuno Faria 1 Paul J. Davis 1 Thomas Tanon 1 Topias Pyykkönen 1 dario curreri ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/54.0.0.md000066400000000000000000000175631521727353300222720ustar00rootroot00000000000000 # Apache DataFusion Python 54.0.0 Changelog This release consists of 51 commits from 8 contributors. See credits at the end of this changelog for more information. **Breaking changes:** - Update datafusion dependency to latest in preparation for DF54 [#1532](https://github.com/apache/datafusion-python/pull/1532) (timsaucer) - feat: enable pickling for Python aggregate and window UDFs [#1545](https://github.com/apache/datafusion-python/pull/1545) (timsaucer) - feat: pass calling SessionContext to Python UDTF callbacks [#1555](https://github.com/apache/datafusion-python/pull/1555) (timsaucer) - feat: accept distinct kwarg on sum and avg [#1556](https://github.com/apache/datafusion-python/pull/1556) (timsaucer) **Implemented enhancements:** - feat: add AI skill to find and improve the Pythonic interface to functions [#1484](https://github.com/apache/datafusion-python/pull/1484) (timsaucer) - feat: enable pickling of most Expr except udaf and udwf [#1544](https://github.com/apache/datafusion-python/pull/1544) (timsaucer) - feat: expose variety of features from DF54 update [#1554](https://github.com/apache/datafusion-python/pull/1554) (timsaucer) - feat: Python UDFs: per-session inlining toggle and strict refusal setting [#1546](https://github.com/apache/datafusion-python/pull/1546) (timsaucer) - feat: create free-threaded python wheels [#1553](https://github.com/apache/datafusion-python/pull/1553) (timsaucer) - feat: expose lambda and higher-order array functions [#1561](https://github.com/apache/datafusion-python/pull/1561) (timsaucer) - feat: import user-defined physical optimizer rules over FFI [#1557](https://github.com/apache/datafusion-python/pull/1557) (timsaucer) - feat: expose SessionContext.copied_config and parse_capacity_limit [#1570](https://github.com/apache/datafusion-python/pull/1570) (timsaucer) - feat: expose array_compact, array_normalize, cosine_distance, inner_product [#1567](https://github.com/apache/datafusion-python/pull/1567) (timsaucer) - feat: expose arrow_field, arrow_try_cast, cast_to_type, with_metadata [#1568](https://github.com/apache/datafusion-python/pull/1568) (timsaucer) - feat: expose spark-compatible functions [#1564](https://github.com/apache/datafusion-python/pull/1564) (timsaucer) - feat: improve pythonic interface on date/time functions [#1563](https://github.com/apache/datafusion-python/pull/1563) (timsaucer) **Fixed bugs:** - fix: type scalar UDF returns as Arrow arrays [#1528](https://github.com/apache/datafusion-python/pull/1528) (BharatDeva) - fix: Skip `fork` and `forkserver` on `win32` [#1566](https://github.com/apache/datafusion-python/pull/1566) (nuno-faria) **Documentation updates:** - docs: enrich module docstrings and add doctest examples [#1498](https://github.com/apache/datafusion-python/pull/1498) (timsaucer) - docs: add README section for AI coding assistants [#1503](https://github.com/apache/datafusion-python/pull/1503) (timsaucer) - docs: add upstream sync process documentation [#1524](https://github.com/apache/datafusion-python/pull/1524) (timsaucer) - docs: document null-handling function arguments [#1527](https://github.com/apache/datafusion-python/pull/1527) (BharatDeva) - docs: user guide + runnable examples for distributing expressions [#1547](https://github.com/apache/datafusion-python/pull/1547) (timsaucer) - docs: convert reStructuredText sources to MyST markdown [#1579](https://github.com/apache/datafusion-python/pull/1579) (timsaucer) **Other:** - Release 53.0.0 [#1491](https://github.com/apache/datafusion-python/pull/1491) (timsaucer) - ci: disable symbol export on Windows verification [#1486](https://github.com/apache/datafusion-python/pull/1486) (timsaucer) - Add Python bindings for accessing ExecutionMetrics [#1381](https://github.com/apache/datafusion-python/pull/1381) (ShreyeshArangath) - Support None comparisons for null expressions [#1489](https://github.com/apache/datafusion-python/pull/1489) (zeel2104) - chore: update release documentation [#1494](https://github.com/apache/datafusion-python/pull/1494) (timsaucer) - Fix error on show() with an explain plan [#1492](https://github.com/apache/datafusion-python/pull/1492) (timsaucer) - Add SKILL.md and enrich package docstring [#1497](https://github.com/apache/datafusion-python/pull/1497) (timsaucer) - minor: fix header on agent SKILL.md file [#1501](https://github.com/apache/datafusion-python/pull/1501) (timsaucer) - tpch examples: rewrite queries idiomatically and embed reference SQL [#1504](https://github.com/apache/datafusion-python/pull/1504) (timsaucer) - Move public skills to a directory to avoid downloading the whole repo [#1519](https://github.com/apache/datafusion-python/pull/1519) (ntjohnson1) - Update user documentation for AI agent skill usage [#1505](https://github.com/apache/datafusion-python/pull/1505) (timsaucer) - build(deps): combined dependabot bumps (Cargo + workflows) [#1534](https://github.com/apache/datafusion-python/pull/1534) (timsaucer) - Add support for logical and physical codecs [#1541](https://github.com/apache/datafusion-python/pull/1541) (timsaucer) - Add details on caching to skill [#1521](https://github.com/apache/datafusion-python/pull/1521) (ntjohnson1) - Bump DataFusion to prepare for DF54 release candidate [#1562](https://github.com/apache/datafusion-python/pull/1562) (timsaucer) - Export `to_datafusion_err` from the util crate root [#1487](https://github.com/apache/datafusion-python/pull/1487) (kosiew) - chore: remove unused PyConfig [#1485](https://github.com/apache/datafusion-python/pull/1485) (timsaucer) - refactor(context): deduplicate register/read option-building logic [#1479](https://github.com/apache/datafusion-python/pull/1479) (mesejo) - Improve documentation site layout [#1578](https://github.com/apache/datafusion-python/pull/1578) (timsaucer) - Allow `DataFrame.aggregate` to accept `None` for no grouping [#1581](https://github.com/apache/datafusion-python/pull/1581) (kosiew) - Update to released DF 54.0.0 [#1588](https://github.com/apache/datafusion-python/pull/1588) (timsaucer) - build(deps): batch dependabot dependency updates [#1589](https://github.com/apache/datafusion-python/pull/1589) (timsaucer) - Deprecate `Expr` temporal part arguments in date extraction and truncation functions [#1587](https://github.com/apache/datafusion-python/pull/1587) (kosiew) - chore: update rust dependencies [#1604](https://github.com/apache/datafusion-python/pull/1604) (timsaucer) - Add deprecation warnings for Expr passed to confirmed literal-only function arguments [#1605](https://github.com/apache/datafusion-python/pull/1605) (kosiew) - chore: resolve audits after DF54.0.0 update to skill [#1608](https://github.com/apache/datafusion-python/pull/1608) (timsaucer) - chore: remove 3.13 freethreaded builds [#1609](https://github.com/apache/datafusion-python/pull/1609) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 39 Tim Saucer 4 kosiew 2 BharatDeva 2 Nick 1 Daniel Mesejo 1 Nuno Faria 1 Shreyesh 1 Zeel Desai ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. apache-datafusion-python-54.0.0/dev/changelog/pre-43.0.0.md000066400000000000000000001570071521727353300230520ustar00rootroot00000000000000 # DataFusion Python Changelog ## [42.0.0](https://github.com/apache/datafusion-python/tree/42.0.0) (2024-10-06) This release consists of 20 commits from 6 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: expose between [#868](https://github.com/apache/datafusion-python/pull/868) (mesejo) - feat: make register_csv accept a list of paths [#883](https://github.com/apache/datafusion-python/pull/883) (mesejo) - feat: expose http object store [#885](https://github.com/apache/datafusion-python/pull/885) (mesejo) **Fixed bugs:** - fix: Calling `count` on a pyarrow dataset results in an error [#843](https://github.com/apache/datafusion-python/pull/843) (Michael-J-Ward) **Other:** - Upgrade datafusion [#867](https://github.com/apache/datafusion-python/pull/867) (emgeee) - Feature/aggregates as windows [#871](https://github.com/apache/datafusion-python/pull/871) (timsaucer) - Fix regression on register_udaf [#878](https://github.com/apache/datafusion-python/pull/878) (timsaucer) - build(deps): upgrade setup-protoc action and protoc version number [#873](https://github.com/apache/datafusion-python/pull/873) (Michael-J-Ward) - build(deps): bump prost-types from 0.13.2 to 0.13.3 [#881](https://github.com/apache/datafusion-python/pull/881) (dependabot[bot]) - build(deps): bump prost from 0.13.2 to 0.13.3 [#882](https://github.com/apache/datafusion-python/pull/882) (dependabot[bot]) - chore: remove XFAIL from passing tests [#884](https://github.com/apache/datafusion-python/pull/884) (Michael-J-Ward) - Add user defined window function support [#880](https://github.com/apache/datafusion-python/pull/880) (timsaucer) - build(deps): bump syn from 2.0.77 to 2.0.79 [#886](https://github.com/apache/datafusion-python/pull/886) (dependabot[bot]) - fix example of reading parquet from s3 [#896](https://github.com/apache/datafusion-python/pull/896) (sir-sigurd) - release-testing [#889](https://github.com/apache/datafusion-python/pull/889) (Michael-J-Ward) - chore(bench): fix create_tables.sql for tpch benchmark [#897](https://github.com/apache/datafusion-python/pull/897) (Michael-J-Ward) - Add physical and logical plan conversion to and from protobuf [#892](https://github.com/apache/datafusion-python/pull/892) (timsaucer) - Feature/instance udfs [#890](https://github.com/apache/datafusion-python/pull/890) (timsaucer) - chore(ci): remove Mambaforge variant from CI [#894](https://github.com/apache/datafusion-python/pull/894) (Michael-J-Ward) - Use OnceLock to store TokioRuntime [#895](https://github.com/apache/datafusion-python/pull/895) (Michael-J-Ward) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 7 Michael J Ward 5 Tim Saucer 3 Daniel Mesejo 3 dependabot[bot] 1 Matt Green 1 Sergey Fedoseev ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. ## [41.0.0](https://github.com/apache/datafusion-python/tree/41.0.0) (2024-09-09) This release consists of 19 commits from 6 contributors. See credits at the end of this changelog for more information. **Implemented enhancements:** - feat: enable list of paths for read_csv [#824](https://github.com/apache/datafusion-python/pull/824) (mesejo) - feat: better exception and message for table not found [#851](https://github.com/apache/datafusion-python/pull/851) (mesejo) - feat: make cast accept built-in Python types [#858](https://github.com/apache/datafusion-python/pull/858) (mesejo) **Other:** - chore: Prepare for 40.0.0 release [#801](https://github.com/apache/datafusion-python/pull/801) (andygrove) - Add typing-extensions dependency to pyproject [#805](https://github.com/apache/datafusion-python/pull/805) (timsaucer) - Upgrade deps to datafusion 41 [#802](https://github.com/apache/datafusion-python/pull/802) (Michael-J-Ward) - Fix SessionContext init with only SessionConfig [#827](https://github.com/apache/datafusion-python/pull/827) (jcrist) - build(deps): upgrade actions/{upload,download}-artifact@v3 to v4 [#829](https://github.com/apache/datafusion-python/pull/829) (Michael-J-Ward) - Run ruff format in CI [#837](https://github.com/apache/datafusion-python/pull/837) (timsaucer) - Add PyCapsule support for Arrow import and export [#825](https://github.com/apache/datafusion-python/pull/825) (timsaucer) - Feature/expose when function [#836](https://github.com/apache/datafusion-python/pull/836) (timsaucer) - Add Window Functions for use with function builder [#808](https://github.com/apache/datafusion-python/pull/808) (timsaucer) - chore: fix typos [#844](https://github.com/apache/datafusion-python/pull/844) (mesejo) - build(ci): use proper mac runners [#841](https://github.com/apache/datafusion-python/pull/841) (Michael-J-Ward) - Set of small features [#839](https://github.com/apache/datafusion-python/pull/839) (timsaucer) - chore: fix docstrings, typos [#852](https://github.com/apache/datafusion-python/pull/852) (mesejo) - chore: Use datafusion re-exported dependencies [#856](https://github.com/apache/datafusion-python/pull/856) (emgeee) - add guidelines on separating python and rust code [#860](https://github.com/apache/datafusion-python/pull/860) (Michael-J-Ward) - Update Aggregate functions to take builder parameters [#859](https://github.com/apache/datafusion-python/pull/859) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 7 Tim Saucer 5 Daniel Mesejo 4 Michael J Ward 1 Andy Grove 1 Jim Crist-Harif 1 Matt Green ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. ## [40.0.0](https://github.com/apache/datafusion-python/tree/40.0.0) (2024-08-09) This release consists of 18 commits from 4 contributors. See credits at the end of this changelog for more information. - Update changelog for 39.0.0 [#742](https://github.com/apache/datafusion-python/pull/742) (andygrove) - build(deps): bump uuid from 1.8.0 to 1.9.1 [#744](https://github.com/apache/datafusion-python/pull/744) (dependabot[bot]) - build(deps): bump mimalloc from 0.1.42 to 0.1.43 [#745](https://github.com/apache/datafusion-python/pull/745) (dependabot[bot]) - build(deps): bump syn from 2.0.67 to 2.0.68 [#746](https://github.com/apache/datafusion-python/pull/746) (dependabot[bot]) - Tsaucer/find window fn [#747](https://github.com/apache/datafusion-python/pull/747) (timsaucer) - Python wrapper classes for all user interfaces [#750](https://github.com/apache/datafusion-python/pull/750) (timsaucer) - Expose array sort [#764](https://github.com/apache/datafusion-python/pull/764) (timsaucer) - Upgrade protobuf and remove GH Action googletest-installer [#773](https://github.com/apache/datafusion-python/pull/773) (Michael-J-Ward) - Upgrade Datafusion 40 [#771](https://github.com/apache/datafusion-python/pull/771) (Michael-J-Ward) - Bugfix: Calling count with None arguments [#768](https://github.com/apache/datafusion-python/pull/768) (timsaucer) - Add in user example that compares a two different approaches to UDFs [#770](https://github.com/apache/datafusion-python/pull/770) (timsaucer) - Add missing exports for wrapper modules [#782](https://github.com/apache/datafusion-python/pull/782) (timsaucer) - Add PyExpr to_variant conversions [#793](https://github.com/apache/datafusion-python/pull/793) (Michael-J-Ward) - Add missing expressions to wrapper export [#795](https://github.com/apache/datafusion-python/pull/795) (timsaucer) - Doc/cross reference [#791](https://github.com/apache/datafusion-python/pull/791) (timsaucer) - Re-Enable `num_centroids` to `approx_percentile_cont` [#798](https://github.com/apache/datafusion-python/pull/798) (Michael-J-Ward) - UDAF process all state variables [#799](https://github.com/apache/datafusion-python/pull/799) (timsaucer) ## Credits Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor. ``` 9 Tim Saucer 4 Michael J Ward 3 dependabot[bot] 2 Andy Grove ``` Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release. ## [39.0.0](https://github.com/apache/datafusion-python/tree/39.0.0) (2024-06-25) **Merged pull requests:** - ci: add substrait feature to linux builds [#720](https://github.com/apache/datafusion-python/pull/720) (Michael-J-Ward) - Docs deploy action [#721](https://github.com/apache/datafusion-python/pull/721) (Michael-J-Ward) - update deps [#723](https://github.com/apache/datafusion-python/pull/723) (Michael-J-Ward) - Upgrade maturin [#725](https://github.com/apache/datafusion-python/pull/725) (Michael-J-Ward) - Upgrade datafusion 39 [#728](https://github.com/apache/datafusion-python/pull/728) (Michael-J-Ward) - use ScalarValue::to_pyarrow to convert to python object [#731](https://github.com/apache/datafusion-python/pull/731) (Michael-J-Ward) - Pyo3 `Bound<'py, T>` api [#734](https://github.com/apache/datafusion-python/pull/734) (Michael-J-Ward) - github test action: drop python 3.7, add python 3.12 [#736](https://github.com/apache/datafusion-python/pull/736) (Michael-J-Ward) - Pyarrow filter pushdowns [#735](https://github.com/apache/datafusion-python/pull/735) (Michael-J-Ward) - build(deps): bump syn from 2.0.66 to 2.0.67 [#738](https://github.com/apache/datafusion-python/pull/738) (dependabot[bot]) - Pyo3 refactorings [#740](https://github.com/apache/datafusion-python/pull/740) (Michael-J-Ward) - UDAF `sum` workaround [#741](https://github.com/apache/datafusion-python/pull/741) (Michael-J-Ward) ## [38.0.1](https://github.com/apache/datafusion-python/tree/38.0.1) (2024-05-25) **Implemented enhancements:** - feat: add python bindings for ends_with function [#693](https://github.com/apache/datafusion-python/pull/693) (richtia) - feat: expose `named_struct` in python [#700](https://github.com/apache/datafusion-python/pull/700) (Michael-J-Ward) **Merged pull requests:** - Add document about basics of working with expressions [#668](https://github.com/apache/datafusion-python/pull/668) (timsaucer) - chore: Update Python release process now that DataFusion is TLP [#674](https://github.com/apache/datafusion-python/pull/674) (andygrove) - Fix Docs [#676](https://github.com/apache/datafusion-python/pull/676) (Michael-J-Ward) - Add examples from TPC-H [#666](https://github.com/apache/datafusion-python/pull/666) (timsaucer) - fix conda nightly builds, attempt 2 [#689](https://github.com/apache/datafusion-python/pull/689) (Michael-J-Ward) - Upgrade to datafusion 38 [#691](https://github.com/apache/datafusion-python/pull/691) (Michael-J-Ward) - chore: update to maturin's recommended project layout for rust/python… [#695](https://github.com/apache/datafusion-python/pull/695) (Michael-J-Ward) - chore: update cargo deps [#698](https://github.com/apache/datafusion-python/pull/698) (Michael-J-Ward) - feat: add python bindings for ends_with function [#693](https://github.com/apache/datafusion-python/pull/693) (richtia) - feat: expose `named_struct` in python [#700](https://github.com/apache/datafusion-python/pull/700) (Michael-J-Ward) - Website fixes [#702](https://github.com/apache/datafusion-python/pull/702) (Michael-J-Ward) ## [37.1.0](https://github.com/apache/datafusion-python/tree/37.1.0) (2024-05-08) **Implemented enhancements:** - feat: add execute_stream and execute_stream_partitioned [#610](https://github.com/apache/datafusion-python/pull/610) (mesejo) **Documentation updates:** - docs: update docs CI to install python-311 requirements [#661](https://github.com/apache/datafusion-python/pull/661) (Michael-J-Ward) **Merged pull requests:** - Switch to Ruff for Python linting [#529](https://github.com/apache/datafusion-python/pull/529) (andygrove) - Remove sql-on-pandas/polars/cudf examples [#602](https://github.com/apache/datafusion-python/pull/602) (andygrove) - build(deps): bump object_store from 0.9.0 to 0.9.1 [#611](https://github.com/apache/datafusion-python/pull/611) (dependabot[bot]) - More missing array funcs [#605](https://github.com/apache/datafusion-python/pull/605) (judahrand) - feat: add execute_stream and execute_stream_partitioned [#610](https://github.com/apache/datafusion-python/pull/610) (mesejo) - build(deps): bump uuid from 1.7.0 to 1.8.0 [#615](https://github.com/apache/datafusion-python/pull/615) (dependabot[bot]) - Bind SQLOptions and relative ctx method #567 [#588](https://github.com/apache/datafusion-python/pull/588) (giacomorebecchi) - bugfix: no panic on empty table [#613](https://github.com/apache/datafusion-python/pull/613) (mesejo) - Expose `register_listing_table` [#618](https://github.com/apache/datafusion-python/pull/618) (henrifroese) - Expose unnest feature [#641](https://github.com/apache/datafusion-python/pull/641) (timsaucer) - Update domain names and paths in asf yaml [#643](https://github.com/apache/datafusion-python/pull/643) (andygrove) - use python 3.11 to publish docs [#645](https://github.com/apache/datafusion-python/pull/645) (andygrove) - docs: update docs CI to install python-311 requirements [#661](https://github.com/apache/datafusion-python/pull/661) (Michael-J-Ward) - Upgrade Datafusion to v37.1.0 [#669](https://github.com/apache/datafusion-python/pull/669) (Michael-J-Ward) ## [36.0.0](https://github.com/apache/datafusion-python/tree/36.0.0) (2024-03-02) **Implemented enhancements:** - feat: Add `flatten` array function [#562](https://github.com/apache/datafusion-python/pull/562) (mobley-trent) **Documentation updates:** - docs: Add ASF attribution [#580](https://github.com/apache/datafusion-python/pull/580) (simicd) **Merged pull requests:** - Allow PyDataFrame to be used from other projects [#582](https://github.com/apache/datafusion-python/pull/582) (andygrove) - docs: Add ASF attribution [#580](https://github.com/apache/datafusion-python/pull/580) (simicd) - Add array functions [#560](https://github.com/apache/datafusion-python/pull/560) (ongchi) - feat: Add `flatten` array function [#562](https://github.com/apache/datafusion-python/pull/562) (mobley-trent) ## [35.0.0](https://github.com/apache/datafusion-python/tree/35.0.0) (2024-01-20) **Merged pull requests:** - build(deps): bump syn from 2.0.41 to 2.0.43 [#559](https://github.com/apache/datafusion-python/pull/559) (dependabot[bot]) - build(deps): bump tokio from 1.35.0 to 1.35.1 [#558](https://github.com/apache/datafusion-python/pull/558) (dependabot[bot]) - build(deps): bump async-trait from 0.1.74 to 0.1.77 [#556](https://github.com/apache/datafusion-python/pull/556) (dependabot[bot]) - build(deps): bump pyo3 from 0.20.0 to 0.20.2 [#557](https://github.com/apache/datafusion-python/pull/557) (dependabot[bot]) ## [34.0.0](https://github.com/apache/datafusion-python/tree/34.0.0) (2023-12-28) **Merged pull requests:** - Adjust visibility of crate private members & Functions [#537](https://github.com/apache/datafusion-python/pull/537) (jdye64) - Update json.rst [#538](https://github.com/apache/datafusion-python/pull/538) (ray-andrew) - Enable mimalloc local_dynamic_tls feature [#540](https://github.com/apache/datafusion-python/pull/540) (jdye64) - Enable substrait feature to be built by default in CI, for nightlies … [#544](https://github.com/apache/datafusion-python/pull/544) (jdye64) ## [33.0.0](https://github.com/apache/datafusion-python/tree/33.0.0) (2023-11-16) **Merged pull requests:** - First pass at getting architectured builds working [#350](https://github.com/apache/datafusion-python/pull/350) (charlesbluca) - Remove libprotobuf dep [#527](https://github.com/apache/datafusion-python/pull/527) (jdye64) ## [32.0.0](https://github.com/apache/datafusion-python/tree/32.0.0) (2023-10-21) **Implemented enhancements:** - feat: expose PyWindowFrame [#509](https://github.com/apache/datafusion-python/pull/509) (dlovell) - add Binary String Functions;encode,decode [#494](https://github.com/apache/datafusion-python/pull/494) (jiangzhx) - add bit_and,bit_or,bit_xor,bool_add,bool_or [#496](https://github.com/apache/datafusion-python/pull/496) (jiangzhx) - add first_value last_value [#498](https://github.com/apache/datafusion-python/pull/498) (jiangzhx) - add regr\_\* functions [#499](https://github.com/apache/datafusion-python/pull/499) (jiangzhx) - Add random missing bindings [#522](https://github.com/apache/datafusion-python/pull/522) (jdye64) - Allow for multiple input files per table instead of a single file [#519](https://github.com/apache/datafusion-python/pull/519) (jdye64) - Add support for window function bindings [#521](https://github.com/apache/datafusion-python/pull/521) (jdye64) **Merged pull requests:** - Prepare 31.0.0 release [#500](https://github.com/apache/datafusion-python/pull/500) (andygrove) - Improve release process documentation [#505](https://github.com/apache/datafusion-python/pull/505) (andygrove) - add Binary String Functions;encode,decode [#494](https://github.com/apache/datafusion-python/pull/494) (jiangzhx) - build(deps): bump mimalloc from 0.1.38 to 0.1.39 [#502](https://github.com/apache/datafusion-python/pull/502) (dependabot[bot]) - build(deps): bump syn from 2.0.32 to 2.0.35 [#503](https://github.com/apache/datafusion-python/pull/503) (dependabot[bot]) - build(deps): bump syn from 2.0.35 to 2.0.37 [#506](https://github.com/apache/datafusion-python/pull/506) (dependabot[bot]) - Use latest DataFusion [#511](https://github.com/apache/datafusion-python/pull/511) (andygrove) - add bit_and,bit_or,bit_xor,bool_add,bool_or [#496](https://github.com/apache/datafusion-python/pull/496) (jiangzhx) - use DataFusion 32 [#515](https://github.com/apache/datafusion-python/pull/515) (andygrove) - add first_value last_value [#498](https://github.com/apache/datafusion-python/pull/498) (jiangzhx) - build(deps): bump regex-syntax from 0.7.5 to 0.8.1 [#517](https://github.com/apache/datafusion-python/pull/517) (dependabot[bot]) - build(deps): bump pyo3-build-config from 0.19.2 to 0.20.0 [#516](https://github.com/apache/datafusion-python/pull/516) (dependabot[bot]) - add regr\_\* functions [#499](https://github.com/apache/datafusion-python/pull/499) (jiangzhx) - Add random missing bindings [#522](https://github.com/apache/datafusion-python/pull/522) (jdye64) - build(deps): bump rustix from 0.38.18 to 0.38.19 [#523](https://github.com/apache/datafusion-python/pull/523) (dependabot[bot]) - Allow for multiple input files per table instead of a single file [#519](https://github.com/apache/datafusion-python/pull/519) (jdye64) - Add support for window function bindings [#521](https://github.com/apache/datafusion-python/pull/521) (jdye64) - Small clippy fix [#524](https://github.com/apache/datafusion-python/pull/524) (andygrove) ## [31.0.0](https://github.com/apache/datafusion-python/tree/31.0.0) (2023-09-12) [Full Changelog](https://github.com/apache/datafusion-python/compare/28.0.0...31.0.0) **Implemented enhancements:** - feat: add case function (#447) [#448](https://github.com/apache/datafusion-python/pull/448) (mesejo) - feat: add compression options [#456](https://github.com/apache/datafusion-python/pull/456) (mesejo) - feat: add register_json [#458](https://github.com/apache/datafusion-python/pull/458) (mesejo) - feat: add basic compression configuration to write_parquet [#459](https://github.com/apache/datafusion-python/pull/459) (mesejo) - feat: add example of reading parquet from s3 [#460](https://github.com/apache/datafusion-python/pull/460) (mesejo) - feat: add register_avro and read_table [#461](https://github.com/apache/datafusion-python/pull/461) (mesejo) - feat: add missing scalar math functions [#465](https://github.com/apache/datafusion-python/pull/465) (mesejo) **Documentation updates:** - docs: include pre-commit hooks section in contributor guide [#455](https://github.com/apache/datafusion-python/pull/455) (mesejo) **Merged pull requests:** - Build Linux aarch64 wheel [#443](https://github.com/apache/datafusion-python/pull/443) (gokselk) - feat: add case function (#447) [#448](https://github.com/apache/datafusion-python/pull/448) (mesejo) - enhancement(docs): Add user guide (#432) [#445](https://github.com/apache/datafusion-python/pull/445) (mesejo) - docs: include pre-commit hooks section in contributor guide [#455](https://github.com/apache/datafusion-python/pull/455) (mesejo) - feat: add compression options [#456](https://github.com/apache/datafusion-python/pull/456) (mesejo) - Upgrade to DF 28.0.0-rc1 [#457](https://github.com/apache/datafusion-python/pull/457) (andygrove) - feat: add register_json [#458](https://github.com/apache/datafusion-python/pull/458) (mesejo) - feat: add basic compression configuration to write_parquet [#459](https://github.com/apache/datafusion-python/pull/459) (mesejo) - feat: add example of reading parquet from s3 [#460](https://github.com/apache/datafusion-python/pull/460) (mesejo) - feat: add register_avro and read_table [#461](https://github.com/apache/datafusion-python/pull/461) (mesejo) - feat: add missing scalar math functions [#465](https://github.com/apache/datafusion-python/pull/465) (mesejo) - build(deps): bump arduino/setup-protoc from 1 to 2 [#452](https://github.com/apache/datafusion-python/pull/452) (dependabot[bot]) - Revert "build(deps): bump arduino/setup-protoc from 1 to 2 (#452)" [#474](https://github.com/apache/datafusion-python/pull/474) (viirya) - Minor: fix wrongly copied function description [#497](https://github.com/apache/datafusion-python/pull/497) (viirya) - Upgrade to Datafusion 31.0.0 [#491](https://github.com/apache/datafusion-python/pull/491) (judahrand) - Add `isnan` and `iszero` [#495](https://github.com/apache/datafusion-python/pull/495) (judahrand) ## 30.0.0 - Skipped due to a breaking change in DataFusion ## 29.0.0 - Skipped ## [28.0.0](https://github.com/apache/datafusion-python/tree/28.0.0) (2023-07-25) **Implemented enhancements:** - feat: expose offset in python API [#437](https://github.com/apache/datafusion-python/pull/437) (cpcloud) **Merged pull requests:** - File based input utils [#433](https://github.com/apache/datafusion-python/pull/433) (jdye64) - Upgrade to 28.0.0-rc1 [#434](https://github.com/apache/datafusion-python/pull/434) (andygrove) - Introduces utility for obtaining SqlTable information from a file like location [#398](https://github.com/apache/datafusion-python/pull/398) (jdye64) - feat: expose offset in python API [#437](https://github.com/apache/datafusion-python/pull/437) (cpcloud) - Use DataFusion 28 [#439](https://github.com/apache/datafusion-python/pull/439) (andygrove) ## [27.0.0](https://github.com/apache/datafusion-python/tree/27.0.0) (2023-07-03) **Merged pull requests:** - LogicalPlan.to_variant() make public [#412](https://github.com/apache/datafusion-python/pull/412) (jdye64) - Prepare 27.0.0 release [#423](https://github.com/apache/datafusion-python/pull/423) (andygrove) ## [26.0.0](https://github.com/apache/datafusion-python/tree/26.0.0) (2023-06-11) [Full Changelog](https://github.com/apache/datafusion-python/compare/25.0.0...26.0.0) **Merged pull requests:** - Add Expr::Case when_then_else support to rex_call_operands function [#388](https://github.com/apache/datafusion-python/pull/388) (jdye64) - Introduce BaseSessionContext abstract class [#390](https://github.com/apache/datafusion-python/pull/390) (jdye64) - CRUD Schema support for `BaseSessionContext` [#392](https://github.com/apache/datafusion-python/pull/392) (jdye64) - CRUD Table support for `BaseSessionContext` [#394](https://github.com/apache/datafusion-python/pull/394) (jdye64) ## [25.0.0](https://github.com/apache/datafusion-python/tree/25.0.0) (2023-05-23) [Full Changelog](https://github.com/apache/datafusion-python/compare/24.0.0...25.0.0) **Merged pull requests:** - Prepare 24.0.0 Release [#376](https://github.com/apache/datafusion-python/pull/376) (andygrove) - build(deps): bump uuid from 1.3.1 to 1.3.2 [#359](https://github.com/apache/datafusion-python/pull/359) (dependabot[bot]) - build(deps): bump mimalloc from 0.1.36 to 0.1.37 [#361](https://github.com/apache/datafusion-python/pull/361) (dependabot[bot]) - build(deps): bump regex-syntax from 0.6.29 to 0.7.1 [#334](https://github.com/apache/datafusion-python/pull/334) (dependabot[bot]) - upgrade maturin to 0.15.1 [#379](https://github.com/apache/datafusion-python/pull/379) (Jimexist) - Expand Expr to include RexType basic support [#378](https://github.com/apache/datafusion-python/pull/378) (jdye64) - Add Python script for generating changelog [#383](https://github.com/apache/datafusion-python/pull/383) (andygrove) ## [24.0.0](https://github.com/apache/datafusion-python/tree/24.0.0) (2023-05-09) [Full Changelog](https://github.com/apache/datafusion-python/compare/23.0.0...24.0.0) **Documentation updates:** - Fix link to user guide [#354](https://github.com/apache/datafusion-python/pull/354) (andygrove) **Merged pull requests:** - Add interface to serialize Substrait plans to Python Bytes. [#344](https://github.com/apache/datafusion-python/pull/344) (kylebrooks-8451) - Add partition_count property to ExecutionPlan. [#346](https://github.com/apache/datafusion-python/pull/346) (kylebrooks-8451) - Remove unsendable from all Rust pyclass types. [#348](https://github.com/apache/datafusion-python/pull/348) (kylebrooks-8451) - Fix link to user guide [#354](https://github.com/apache/datafusion-python/pull/354) (andygrove) - Fix SessionContext execute. [#353](https://github.com/apache/datafusion-python/pull/353) (kylebrooks-8451) - Pub mod expr in lib.rs [#357](https://github.com/apache/datafusion-python/pull/357) (jdye64) - Add benchmark derived from TPC-H [#355](https://github.com/apache/datafusion-python/pull/355) (andygrove) - Add db-benchmark [#365](https://github.com/apache/datafusion-python/pull/365) (andygrove) - First pass of documentation in mdBook [#364](https://github.com/apache/datafusion-python/pull/364) (MrPowers) - Add 'pub' and '#[pyo3(get, set)]' to DataTypeMap [#371](https://github.com/apache/datafusion-python/pull/371) (jdye64) - Fix db-benchmark [#369](https://github.com/apache/datafusion-python/pull/369) (andygrove) - Docs explaining how to view query plans [#373](https://github.com/apache/datafusion-python/pull/373) (andygrove) - Improve db-benchmark [#372](https://github.com/apache/datafusion-python/pull/372) (andygrove) - Make expr member of PyExpr public [#375](https://github.com/apache/datafusion-python/pull/375) (jdye64) ## [23.0.0](https://github.com/apache/datafusion-python/tree/23.0.0) (2023-04-23) [Full Changelog](https://github.com/apache/datafusion-python/compare/22.0.0...23.0.0) **Merged pull requests:** - Improve API docs, README, and examples for configuring context [#321](https://github.com/apache/datafusion-python/pull/321) (andygrove) - Osx build linker args [#330](https://github.com/apache/datafusion-python/pull/330) (jdye64) - Add requirements file for python 3.11 [#332](https://github.com/apache/datafusion-python/pull/332) (r4ntix) - mac arm64 build [#338](https://github.com/apache/datafusion-python/pull/338) (andygrove) - Add conda.yaml baseline workflow file [#281](https://github.com/apache/datafusion-python/pull/281) (jdye64) - Prepare for 23.0.0 release [#335](https://github.com/apache/datafusion-python/pull/335) (andygrove) - Reuse the Tokio Runtime [#341](https://github.com/apache/datafusion-python/pull/341) (kylebrooks-8451) ## [22.0.0](https://github.com/apache/datafusion-python/tree/22.0.0) (2023-04-10) [Full Changelog](https://github.com/apache/datafusion-python/compare/21.0.0...22.0.0) **Merged pull requests:** - Fix invalid build yaml [#308](https://github.com/apache/datafusion-python/pull/308) (andygrove) - Try fix release build [#309](https://github.com/apache/datafusion-python/pull/309) (andygrove) - Fix release build [#310](https://github.com/apache/datafusion-python/pull/310) (andygrove) - Enable datafusion-substrait protoc feature, to remove compile-time dependency on protoc [#312](https://github.com/apache/datafusion-python/pull/312) (andygrove) - Fix Mac/Win release builds in CI [#313](https://github.com/apache/datafusion-python/pull/313) (andygrove) - install protoc in docs workflow [#314](https://github.com/apache/datafusion-python/pull/314) (andygrove) - Fix documentation generation in CI [#315](https://github.com/apache/datafusion-python/pull/315) (andygrove) - Source wheel fix [#319](https://github.com/apache/datafusion-python/pull/319) (andygrove) ## [21.0.0](https://github.com/apache/datafusion-python/tree/21.0.0) (2023-03-30) [Full Changelog](https://github.com/apache/datafusion-python/compare/20.0.0...21.0.0) **Merged pull requests:** - minor: Fix minor warning on unused import [#289](https://github.com/apache/datafusion-python/pull/289) (viirya) - feature: Implement `describe()` method [#293](https://github.com/apache/datafusion-python/pull/293) (simicd) - fix: Printed results not visible in debugger & notebooks [#296](https://github.com/apache/datafusion-python/pull/296) (simicd) - add package.include and remove wildcard dependency [#295](https://github.com/apache/datafusion-python/pull/295) (andygrove) - Update main branch name in docs workflow [#303](https://github.com/apache/datafusion-python/pull/303) (andygrove) - Upgrade to DF 21 [#301](https://github.com/apache/datafusion-python/pull/301) (andygrove) ## [20.0.0](https://github.com/apache/datafusion-python/tree/20.0.0) (2023-03-17) [Full Changelog](https://github.com/apache/datafusion-python/compare/0.8.0...20.0.0) **Implemented enhancements:** - Empty relation bindings [#208](https://github.com/apache/datafusion-python/pull/208) (jdye64) - wrap display_name and canonical_name functions [#214](https://github.com/apache/datafusion-python/pull/214) (jdye64) - Add PyAlias bindings [#216](https://github.com/apache/datafusion-python/pull/216) (jdye64) - Add bindings for scalar_variable [#218](https://github.com/apache/datafusion-python/pull/218) (jdye64) - Bindings for LIKE type expressions [#220](https://github.com/apache/datafusion-python/pull/220) (jdye64) - Bool expr bindings [#223](https://github.com/apache/datafusion-python/pull/223) (jdye64) - Between bindings [#229](https://github.com/apache/datafusion-python/pull/229) (jdye64) - Add bindings for GetIndexedField [#227](https://github.com/apache/datafusion-python/pull/227) (jdye64) - Add bindings for case, cast, and trycast [#232](https://github.com/apache/datafusion-python/pull/232) (jdye64) - add remaining expr bindings [#233](https://github.com/apache/datafusion-python/pull/233) (jdye64) - feature: Additional export methods [#236](https://github.com/apache/datafusion-python/pull/236) (simicd) - Add Python wrapper for LogicalPlan::Union [#240](https://github.com/apache/datafusion-python/pull/240) (iajoiner) - feature: Create dataframe from pandas, polars, dictionary, list or pyarrow Table [#242](https://github.com/apache/datafusion-python/pull/242) (simicd) - Add Python wrappers for `LogicalPlan::Join` and `LogicalPlan::CrossJoin` [#246](https://github.com/apache/datafusion-python/pull/246) (iajoiner) - feature: Set table name from ctx functions [#260](https://github.com/apache/datafusion-python/pull/260) (simicd) - Explain bindings [#264](https://github.com/apache/datafusion-python/pull/264) (jdye64) - Extension bindings [#266](https://github.com/apache/datafusion-python/pull/266) (jdye64) - Subquery alias bindings [#269](https://github.com/apache/datafusion-python/pull/269) (jdye64) - Create memory table [#271](https://github.com/apache/datafusion-python/pull/271) (jdye64) - Create view bindings [#273](https://github.com/apache/datafusion-python/pull/273) (jdye64) - Re-export Datafusion dependencies [#277](https://github.com/apache/datafusion-python/pull/277) (jdye64) - Distinct bindings [#275](https://github.com/apache/datafusion-python/pull/275) (jdye64) - Drop table bindings [#283](https://github.com/apache/datafusion-python/pull/283) (jdye64) - Bindings for LogicalPlan::Repartition [#285](https://github.com/apache/datafusion-python/pull/285) (jdye64) - Expand Rust return type support for Arrow DataTypes in ScalarValue [#287](https://github.com/apache/datafusion-python/pull/287) (jdye64) **Documentation updates:** - docs: Example of calling Python UDF & UDAF in SQL [#258](https://github.com/apache/datafusion-python/pull/258) (simicd) **Merged pull requests:** - Minor docs updates [#210](https://github.com/apache/datafusion-python/pull/210) (andygrove) - Empty relation bindings [#208](https://github.com/apache/datafusion-python/pull/208) (jdye64) - wrap display_name and canonical_name functions [#214](https://github.com/apache/datafusion-python/pull/214) (jdye64) - Add PyAlias bindings [#216](https://github.com/apache/datafusion-python/pull/216) (jdye64) - Add bindings for scalar_variable [#218](https://github.com/apache/datafusion-python/pull/218) (jdye64) - Bindings for LIKE type expressions [#220](https://github.com/apache/datafusion-python/pull/220) (jdye64) - Bool expr bindings [#223](https://github.com/apache/datafusion-python/pull/223) (jdye64) - Between bindings [#229](https://github.com/apache/datafusion-python/pull/229) (jdye64) - Add bindings for GetIndexedField [#227](https://github.com/apache/datafusion-python/pull/227) (jdye64) - Add bindings for case, cast, and trycast [#232](https://github.com/apache/datafusion-python/pull/232) (jdye64) - add remaining expr bindings [#233](https://github.com/apache/datafusion-python/pull/233) (jdye64) - Pre-commit hooks [#228](https://github.com/apache/datafusion-python/pull/228) (jdye64) - Implement new release process [#149](https://github.com/apache/datafusion-python/pull/149) (andygrove) - feature: Additional export methods [#236](https://github.com/apache/datafusion-python/pull/236) (simicd) - Add Python wrapper for LogicalPlan::Union [#240](https://github.com/apache/datafusion-python/pull/240) (iajoiner) - feature: Create dataframe from pandas, polars, dictionary, list or pyarrow Table [#242](https://github.com/apache/datafusion-python/pull/242) (simicd) - Fix release instructions [#238](https://github.com/apache/datafusion-python/pull/238) (andygrove) - Add Python wrappers for `LogicalPlan::Join` and `LogicalPlan::CrossJoin` [#246](https://github.com/apache/datafusion-python/pull/246) (iajoiner) - docs: Example of calling Python UDF & UDAF in SQL [#258](https://github.com/apache/datafusion-python/pull/258) (simicd) - feature: Set table name from ctx functions [#260](https://github.com/apache/datafusion-python/pull/260) (simicd) - Upgrade to DataFusion 19 [#262](https://github.com/apache/datafusion-python/pull/262) (andygrove) - Explain bindings [#264](https://github.com/apache/datafusion-python/pull/264) (jdye64) - Extension bindings [#266](https://github.com/apache/datafusion-python/pull/266) (jdye64) - Subquery alias bindings [#269](https://github.com/apache/datafusion-python/pull/269) (jdye64) - Create memory table [#271](https://github.com/apache/datafusion-python/pull/271) (jdye64) - Create view bindings [#273](https://github.com/apache/datafusion-python/pull/273) (jdye64) - Re-export Datafusion dependencies [#277](https://github.com/apache/datafusion-python/pull/277) (jdye64) - Distinct bindings [#275](https://github.com/apache/datafusion-python/pull/275) (jdye64) - build(deps): bump actions/checkout from 2 to 3 [#244](https://github.com/apache/datafusion-python/pull/244) (dependabot[bot]) - build(deps): bump actions/upload-artifact from 2 to 3 [#245](https://github.com/apache/datafusion-python/pull/245) (dependabot[bot]) - build(deps): bump actions/download-artifact from 2 to 3 [#243](https://github.com/apache/datafusion-python/pull/243) (dependabot[bot]) - Use DataFusion 20 [#278](https://github.com/apache/datafusion-python/pull/278) (andygrove) - Drop table bindings [#283](https://github.com/apache/datafusion-python/pull/283) (jdye64) - Bindings for LogicalPlan::Repartition [#285](https://github.com/apache/datafusion-python/pull/285) (jdye64) - Expand Rust return type support for Arrow DataTypes in ScalarValue [#287](https://github.com/apache/datafusion-python/pull/287) (jdye64) ## [0.8.0](https://github.com/apache/datafusion-python/tree/0.8.0) (2023-02-22) [Full Changelog](https://github.com/apache/datafusion-python/compare/0.8.0-rc1...0.8.0) **Implemented enhancements:** - Add support for cuDF physical execution engine [\#202](https://github.com/apache/datafusion-python/issues/202) - Make it easier to create a Pandas dataframe from DataFusion query results [\#139](https://github.com/apache/datafusion-python/issues/139) **Fixed bugs:** - Build error: could not compile `thiserror` due to 2 previous errors [\#69](https://github.com/apache/datafusion-python/issues/69) **Closed issues:** - Integrate with the new `object_store` crate [\#22](https://github.com/apache/datafusion-python/issues/22) **Merged pull requests:** - Update README in preparation for 0.8 release [\#206](https://github.com/apache/datafusion-python/pull/206) ([andygrove](https://github.com/andygrove)) - Add support for cudf as a physical execution engine [\#205](https://github.com/apache/datafusion-python/pull/205) ([jdye64](https://github.com/jdye64)) - Run `maturin develop` instead of `cargo build` in verification script [\#200](https://github.com/apache/datafusion-python/pull/200) ([andygrove](https://github.com/andygrove)) - Add tests for recently added functionality [\#199](https://github.com/apache/datafusion-python/pull/199) ([andygrove](https://github.com/andygrove)) - Implement `to_pandas()` [\#197](https://github.com/apache/datafusion-python/pull/197) ([simicd](https://github.com/simicd)) - Add Python wrapper for LogicalPlan::Sort [\#196](https://github.com/apache/datafusion-python/pull/196) ([andygrove](https://github.com/andygrove)) - Add Python wrapper for LogicalPlan::Aggregate [\#195](https://github.com/apache/datafusion-python/pull/195) ([andygrove](https://github.com/andygrove)) - Add Python wrapper for LogicalPlan::Limit [\#193](https://github.com/apache/datafusion-python/pull/193) ([andygrove](https://github.com/andygrove)) - Add Python wrapper for LogicalPlan::Filter [\#192](https://github.com/apache/datafusion-python/pull/192) ([andygrove](https://github.com/andygrove)) - Add experimental support for executing SQL with Polars and Pandas [\#190](https://github.com/apache/datafusion-python/pull/190) ([andygrove](https://github.com/andygrove)) - Update changelog for 0.8 release [\#188](https://github.com/apache/datafusion-python/pull/188) ([andygrove](https://github.com/andygrove)) - Add ability to execute ExecutionPlan and get a stream of RecordBatch [\#186](https://github.com/apache/datafusion-python/pull/186) ([andygrove](https://github.com/andygrove)) - Dffield bindings [\#185](https://github.com/apache/datafusion-python/pull/185) ([jdye64](https://github.com/jdye64)) - Add bindings for DFSchema [\#183](https://github.com/apache/datafusion-python/pull/183) ([jdye64](https://github.com/jdye64)) - test: Window functions [\#182](https://github.com/apache/datafusion-python/pull/182) ([simicd](https://github.com/simicd)) - Add bindings for Projection [\#180](https://github.com/apache/datafusion-python/pull/180) ([jdye64](https://github.com/jdye64)) - Table scan bindings [\#178](https://github.com/apache/datafusion-python/pull/178) ([jdye64](https://github.com/jdye64)) - Make session configurable [\#176](https://github.com/apache/datafusion-python/pull/176) ([andygrove](https://github.com/andygrove)) - Upgrade to DataFusion 18.0.0 [\#175](https://github.com/apache/datafusion-python/pull/175) ([andygrove](https://github.com/andygrove)) - Use latest DataFusion rev in preparation for DF 18 release [\#174](https://github.com/apache/datafusion-python/pull/174) ([andygrove](https://github.com/andygrove)) - Arrow type bindings [\#173](https://github.com/apache/datafusion-python/pull/173) ([jdye64](https://github.com/jdye64)) - Pyo3 bump [\#171](https://github.com/apache/datafusion-python/pull/171) ([jdye64](https://github.com/jdye64)) - feature: Add additional aggregation functions [\#170](https://github.com/apache/datafusion-python/pull/170) ([simicd](https://github.com/simicd)) - Make from_substrait_plan return DataFrame instead of LogicalPlan [\#164](https://github.com/apache/datafusion-python/pull/164) ([andygrove](https://github.com/andygrove)) - feature: Implement count method [\#163](https://github.com/apache/datafusion-python/pull/163) ([simicd](https://github.com/simicd)) - CI Fixes [\#162](https://github.com/apache/datafusion-python/pull/162) ([jdye64](https://github.com/jdye64)) - Upgrade to DataFusion 17 [\#160](https://github.com/apache/datafusion-python/pull/160) ([andygrove](https://github.com/andygrove)) - feature: Improve string representation of datafusion classes [\#159](https://github.com/apache/datafusion-python/pull/159) ([simicd](https://github.com/simicd)) - Make PyExecutionPlan.plan public [\#156](https://github.com/apache/datafusion-python/pull/156) ([andygrove](https://github.com/andygrove)) - Expose methods on logical and execution plans [\#155](https://github.com/apache/datafusion-python/pull/155) ([andygrove](https://github.com/andygrove)) - Fix clippy for new Rust version [\#154](https://github.com/apache/datafusion-python/pull/154) ([andygrove](https://github.com/andygrove)) - Add DataFrame methods for accessing plans [\#153](https://github.com/apache/datafusion-python/pull/153) ([andygrove](https://github.com/andygrove)) - Use DataFusion rev 5238e8c97f998b4d2cb9fab85fb182f325a1a7fb [\#150](https://github.com/apache/datafusion-python/pull/150) ([andygrove](https://github.com/andygrove)) - build\(deps\): bump async-trait from 0.1.61 to 0.1.62 [\#148](https://github.com/apache/datafusion-python/pull/148) ([dependabot[bot]](https://github.com/apps/dependabot)) - Rename default branch from master to main [\#147](https://github.com/apache/datafusion-python/pull/147) ([andygrove](https://github.com/andygrove)) - Substrait bindings [\#145](https://github.com/apache/datafusion-python/pull/145) ([jdye64](https://github.com/jdye64)) - build\(deps\): bump uuid from 0.8.2 to 1.2.2 [\#143](https://github.com/apache/datafusion-python/pull/143) ([dependabot[bot]](https://github.com/apps/dependabot)) - Prepare for 0.8.0 release [\#141](https://github.com/apache/datafusion-python/pull/141) ([andygrove](https://github.com/andygrove)) - Improve README and add more examples [\#137](https://github.com/apache/datafusion-python/pull/137) ([andygrove](https://github.com/andygrove)) - test: Expand tests for built-in functions [\#129](https://github.com/apache/datafusion-python/pull/129) ([simicd](https://github.com/simicd)) - build\(deps\): bump object_store from 0.5.2 to 0.5.3 [\#126](https://github.com/apache/datafusion-python/pull/126) ([dependabot[bot]](https://github.com/apps/dependabot)) - build\(deps\): bump mimalloc from 0.1.32 to 0.1.34 [\#125](https://github.com/apache/datafusion-python/pull/125) ([dependabot[bot]](https://github.com/apps/dependabot)) - Introduce conda directory containing datafusion-dev.yaml conda enviro… [\#124](https://github.com/apache/datafusion-python/pull/124) ([jdye64](https://github.com/jdye64)) - build\(deps\): bump bzip2 from 0.4.3 to 0.4.4 [\#121](https://github.com/apache/datafusion-python/pull/121) ([dependabot[bot]](https://github.com/apps/dependabot)) - build\(deps\): bump tokio from 1.23.0 to 1.24.1 [\#119](https://github.com/apache/datafusion-python/pull/119) ([dependabot[bot]](https://github.com/apps/dependabot)) - build\(deps\): bump async-trait from 0.1.60 to 0.1.61 [\#118](https://github.com/apache/datafusion-python/pull/118) ([dependabot[bot]](https://github.com/apps/dependabot)) - Upgrade to DataFusion 16.0.0 [\#115](https://github.com/apache/datafusion-python/pull/115) ([andygrove](https://github.com/andygrove)) - Bump async-trait from 0.1.57 to 0.1.60 [\#114](https://github.com/apache/datafusion-python/pull/114) ([dependabot[bot]](https://github.com/apps/dependabot)) - Bump object_store from 0.5.1 to 0.5.2 [\#112](https://github.com/apache/datafusion-python/pull/112) ([dependabot[bot]](https://github.com/apps/dependabot)) - Bump tokio from 1.21.2 to 1.23.0 [\#109](https://github.com/apache/datafusion-python/pull/109) ([dependabot[bot]](https://github.com/apps/dependabot)) - Add entries for publishing production \(asf-site\) and staging docs [\#107](https://github.com/apache/datafusion-python/pull/107) ([martin-g](https://github.com/martin-g)) - Add a workflow that builds the docs and deploys them at staged or production [\#104](https://github.com/apache/datafusion-python/pull/104) ([martin-g](https://github.com/martin-g)) - Upgrade to DataFusion 15.0.0 [\#103](https://github.com/apache/datafusion-python/pull/103) ([andygrove](https://github.com/andygrove)) - build\(deps\): bump futures from 0.3.24 to 0.3.25 [\#102](https://github.com/apache/datafusion-python/pull/102) ([dependabot[bot]](https://github.com/apps/dependabot)) - build\(deps\): bump pyo3 from 0.17.2 to 0.17.3 [\#101](https://github.com/apache/datafusion-python/pull/101) ([dependabot[bot]](https://github.com/apps/dependabot)) - build\(deps\): bump mimalloc from 0.1.30 to 0.1.32 [\#98](https://github.com/apache/datafusion-python/pull/98) ([dependabot[bot]](https://github.com/apps/dependabot)) - build\(deps\): bump rand from 0.7.3 to 0.8.5 [\#97](https://github.com/apache/datafusion-python/pull/97) ([dependabot[bot]](https://github.com/apps/dependabot)) - Fix GitHub actions warnings [\#95](https://github.com/apache/datafusion-python/pull/95) ([martin-g](https://github.com/martin-g)) - Fixes \#81 - Add CI workflow for source distribution [\#93](https://github.com/apache/datafusion-python/pull/93) ([martin-g](https://github.com/martin-g)) - post-release updates [\#91](https://github.com/apache/datafusion-python/pull/91) ([andygrove](https://github.com/andygrove)) - Build for manylinux 2014 [\#88](https://github.com/apache/datafusion-python/pull/88) ([martin-g](https://github.com/martin-g)) - update release readme tag [\#86](https://github.com/apache/datafusion-python/pull/86) ([Jimexist](https://github.com/Jimexist)) - Upgrade Maturin to 0.14.2 [\#85](https://github.com/apache/datafusion-python/pull/85) ([martin-g](https://github.com/martin-g)) - Update release instructions [\#83](https://github.com/apache/datafusion-python/pull/83) ([andygrove](https://github.com/andygrove)) - \[Functions\] - Add python function binding to `functions` [\#73](https://github.com/apache/datafusion-python/pull/73) ([francis-du](https://github.com/francis-du)) ## [0.8.0-rc1](https://github.com/apache/datafusion-python/tree/0.8.0-rc1) (2023-02-17) [Full Changelog](https://github.com/apache/datafusion-python/compare/0.7.0-rc2...0.8.0-rc1) **Implemented enhancements:** - Add bindings for datafusion_common::DFField [\#184](https://github.com/apache/datafusion-python/issues/184) - Add bindings for DFSchema/DFSchemaRef [\#181](https://github.com/apache/datafusion-python/issues/181) - Add bindings for datafusion_expr Projection [\#179](https://github.com/apache/datafusion-python/issues/179) - Add bindings for `TableScan` struct from `datafusion_expr::TableScan` [\#177](https://github.com/apache/datafusion-python/issues/177) - Add a "mapping" struct for types [\#172](https://github.com/apache/datafusion-python/issues/172) - Improve string representation of datafusion classes \(dataframe, context, expression, ...\) [\#158](https://github.com/apache/datafusion-python/issues/158) - Add DataFrame count method [\#151](https://github.com/apache/datafusion-python/issues/151) - \[REQUEST\] Github Actions Improvements [\#146](https://github.com/apache/datafusion-python/issues/146) - Change default branch name from master to main [\#144](https://github.com/apache/datafusion-python/issues/144) - Bump pyo3 to 0.18.0 [\#140](https://github.com/apache/datafusion-python/issues/140) - Add script for Python linting [\#134](https://github.com/apache/datafusion-python/issues/134) - Add Python bindings for substrait module [\#132](https://github.com/apache/datafusion-python/issues/132) - Expand unit tests for built-in functions [\#128](https://github.com/apache/datafusion-python/issues/128) - support creating arrow-datafusion-python conda environment [\#122](https://github.com/apache/datafusion-python/issues/122) - Build Python source distribution in GitHub workflow [\#81](https://github.com/apache/datafusion-python/issues/81) - EPIC: Add all functions to python binding `functions` [\#72](https://github.com/apache/datafusion-python/issues/72) **Fixed bugs:** - Build is broken [\#161](https://github.com/apache/datafusion-python/issues/161) - Out of memory when sorting [\#157](https://github.com/apache/datafusion-python/issues/157) - window_lead test appears to be non-deterministic [\#135](https://github.com/apache/datafusion-python/issues/135) - Reading csv does not work [\#130](https://github.com/apache/datafusion-python/issues/130) - Github actions produce a lot of warnings [\#94](https://github.com/apache/datafusion-python/issues/94) - ASF source release tarball has wrong directory name [\#90](https://github.com/apache/datafusion-python/issues/90) - Python Release Build failing after upgrading to maturin 14.2 [\#87](https://github.com/apache/datafusion-python/issues/87) - Maturin build hangs on Linux ARM64 [\#84](https://github.com/apache/datafusion-python/issues/84) - Cannot install on Mac M1 from source tarball from testpypi [\#82](https://github.com/apache/datafusion-python/issues/82) - ImportPathMismatchError when running pytest locally [\#77](https://github.com/apache/datafusion-python/issues/77) **Closed issues:** - Publish documentation for Python bindings [\#39](https://github.com/apache/datafusion-python/issues/39) - Add Python binding for `approx_median` [\#32](https://github.com/apache/datafusion-python/issues/32) - Release version 0.7.0 [\#7](https://github.com/apache/datafusion-python/issues/7) ## [0.7.0-rc2](https://github.com/apache/datafusion-python/tree/0.7.0-rc2) (2022-11-26) [Full Changelog](https://github.com/apache/datafusion-python/compare/0.7.0...0.7.0-rc2) ## [Unreleased](https://github.com/datafusion-contrib/datafusion-python/tree/HEAD) [Full Changelog](https://github.com/datafusion-contrib/datafusion-python/compare/0.5.1...HEAD) **Merged pull requests:** - use \_\_getitem\_\_ for df column selection [\#41](https://github.com/datafusion-contrib/datafusion-python/pull/41) ([Jimexist](https://github.com/Jimexist)) - fix demo in readme [\#40](https://github.com/datafusion-contrib/datafusion-python/pull/40) ([Jimexist](https://github.com/Jimexist)) - Implement select_columns [\#39](https://github.com/datafusion-contrib/datafusion-python/pull/39) ([andygrove](https://github.com/andygrove)) - update readme and changelog [\#38](https://github.com/datafusion-contrib/datafusion-python/pull/38) ([Jimexist](https://github.com/Jimexist)) - Add PyDataFrame.explain [\#36](https://github.com/datafusion-contrib/datafusion-python/pull/36) ([andygrove](https://github.com/andygrove)) - Release 0.5.0 [\#34](https://github.com/datafusion-contrib/datafusion-python/pull/34) ([Jimexist](https://github.com/Jimexist)) - disable nightly in workflow [\#33](https://github.com/datafusion-contrib/datafusion-python/pull/33) ([Jimexist](https://github.com/Jimexist)) - update requirements to 37 and 310, update readme [\#32](https://github.com/datafusion-contrib/datafusion-python/pull/32) ([Jimexist](https://github.com/Jimexist)) - Add custom global allocator [\#30](https://github.com/datafusion-contrib/datafusion-python/pull/30) ([matthewmturner](https://github.com/matthewmturner)) - Remove pandas dependency [\#25](https://github.com/datafusion-contrib/datafusion-python/pull/25) ([matthewmturner](https://github.com/matthewmturner)) - upgrade datafusion and pyo3 [\#20](https://github.com/datafusion-contrib/datafusion-python/pull/20) ([Jimexist](https://github.com/Jimexist)) - update maturin 0.12+ [\#17](https://github.com/datafusion-contrib/datafusion-python/pull/17) ([Jimexist](https://github.com/Jimexist)) - Update README.md [\#16](https://github.com/datafusion-contrib/datafusion-python/pull/16) ([Jimexist](https://github.com/Jimexist)) - apply cargo clippy --fix [\#15](https://github.com/datafusion-contrib/datafusion-python/pull/15) ([Jimexist](https://github.com/Jimexist)) - update test workflow to include rust clippy and check [\#14](https://github.com/datafusion-contrib/datafusion-python/pull/14) ([Jimexist](https://github.com/Jimexist)) - use maturin 0.12.6 [\#13](https://github.com/datafusion-contrib/datafusion-python/pull/13) ([Jimexist](https://github.com/Jimexist)) - apply cargo fmt [\#12](https://github.com/datafusion-contrib/datafusion-python/pull/12) ([Jimexist](https://github.com/Jimexist)) - use stable not nightly [\#11](https://github.com/datafusion-contrib/datafusion-python/pull/11) ([Jimexist](https://github.com/Jimexist)) - ci: test against more compilers, setup clippy and fix clippy lints [\#9](https://github.com/datafusion-contrib/datafusion-python/pull/9) ([cpcloud](https://github.com/cpcloud)) - Fix use of importlib.metadata and unify requirements.txt [\#8](https://github.com/datafusion-contrib/datafusion-python/pull/8) ([cpcloud](https://github.com/cpcloud)) - Ship the Cargo.lock file in the source distribution [\#7](https://github.com/datafusion-contrib/datafusion-python/pull/7) ([cpcloud](https://github.com/cpcloud)) - add \_\_version\_\_ attribute to datafusion object [\#3](https://github.com/datafusion-contrib/datafusion-python/pull/3) ([tfeda](https://github.com/tfeda)) - fix ci by fixing directories [\#2](https://github.com/datafusion-contrib/datafusion-python/pull/2) ([Jimexist](https://github.com/Jimexist)) - setup workflow [\#1](https://github.com/datafusion-contrib/datafusion-python/pull/1) ([Jimexist](https://github.com/Jimexist)) ## [0.5.1](https://github.com/datafusion-contrib/datafusion-python/tree/0.5.1) (2022-03-15) [Full Changelog](https://github.com/datafusion-contrib/datafusion-python/compare/0.5.1-rc1...0.5.1) ## [0.5.1-rc1](https://github.com/datafusion-contrib/datafusion-python/tree/0.5.1-rc1) (2022-03-15) [Full Changelog](https://github.com/datafusion-contrib/datafusion-python/compare/0.5.0...0.5.1-rc1) ## [0.5.0](https://github.com/datafusion-contrib/datafusion-python/tree/0.5.0) (2022-03-10) [Full Changelog](https://github.com/datafusion-contrib/datafusion-python/compare/0.5.0-rc2...0.5.0) ## [0.5.0-rc2](https://github.com/datafusion-contrib/datafusion-python/tree/0.5.0-rc2) (2022-03-10) [Full Changelog](https://github.com/datafusion-contrib/datafusion-python/compare/0.5.0-rc1...0.5.0-rc2) **Closed issues:** - Add support for Ballista [\#37](https://github.com/datafusion-contrib/datafusion-python/issues/37) - Implement DataFrame.explain [\#35](https://github.com/datafusion-contrib/datafusion-python/issues/35) ## [0.5.0-rc1](https://github.com/datafusion-contrib/datafusion-python/tree/0.5.0-rc1) (2022-03-09) [Full Changelog](https://github.com/datafusion-contrib/datafusion-python/compare/4c98b8e9c3c3f8e2e6a8f2d1ffcfefda344c4680...0.5.0-rc1) **Closed issues:** - Investigate exposing additional optimizations [\#28](https://github.com/datafusion-contrib/datafusion-python/issues/28) - Use custom allocator in Python build [\#27](https://github.com/datafusion-contrib/datafusion-python/issues/27) - Why is pandas a requirement? [\#24](https://github.com/datafusion-contrib/datafusion-python/issues/24) - Unable to build [\#18](https://github.com/datafusion-contrib/datafusion-python/issues/18) - Setup CI against multiple Python version [\#6](https://github.com/datafusion-contrib/datafusion-python/issues/6) apache-datafusion-python-54.0.0/dev/check_crates_patch.py000066400000000000000000000037651521727353300234430ustar00rootroot00000000000000#!/usr/bin/env python3 # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Check that no Cargo.toml files contain [patch.crates-io] entries. Release builds must not depend on patched crates. During development it is common to temporarily patch crates-io dependencies, but those patches must be removed before creating a release. An empty [patch.crates-io] section is allowed. """ import sys from pathlib import Path import tomllib def main() -> int: errors: list[str] = [] for cargo_toml in sorted(Path().rglob("Cargo.toml")): if "target" in cargo_toml.parts: continue with Path.open(cargo_toml, "rb") as f: data = tomllib.load(f) patch = data.get("patch", {}).get("crates-io", {}) if patch: errors.append(str(cargo_toml)) for name, spec in patch.items(): errors.append(f" {name} = {spec}") if errors: print("ERROR: Release builds must not contain [patch.crates-io] entries.") print() for line in errors: print(line) print() print("Remove all [patch.crates-io] entries before creating a release.") return 1 print("OK: No [patch.crates-io] entries found.") return 0 if __name__ == "__main__": sys.exit(main()) apache-datafusion-python-54.0.0/dev/clean.sh000077500000000000000000000033441521727353300207060ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # # This cleans up the project by removing build artifacts and other generated files. # Function to remove a directory and print the action remove_dir() { if [ -d "$1" ]; then echo "Removing directory: $1" rm -rf "$1" fi } # Function to remove a file and print the action remove_file() { if [ -f "$1" ]; then echo "Removing file: $1" rm -f "$1" fi } # Remove .pytest_cache directory remove_dir .pytest_cache/ # Remove target directory remove_dir target/ # Remove any __pycache__ directories find python/ -type d -name "__pycache__" -print | while read -r dir; do remove_dir "$dir" done # Remove pytest-coverage.lcov file # remove_file .coverage # remove_file pytest-coverage.lcov # Remove rust-coverage.lcov file # remove_file rust-coverage.lcov # Remove pyo3 files find python/ -type f -name '_internal.*.so' -print | while read -r file; do remove_file "$file" done echo "Cleanup complete."apache-datafusion-python-54.0.0/dev/create_license.py000066400000000000000000000314471521727353300226110ustar00rootroot00000000000000#!/usr/bin/python # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # This file is a mirror of https://github.com/apache/arrow-datafusion/blob/master/dev/create_license.py import json import subprocess from pathlib import Path data = subprocess.check_output( [ "cargo-license", "--avoid-build-deps", "--avoid-dev-deps", "--do-not-bundle", "--json", ] ) data = json.loads(data) result = """ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. """ result += "\n------------------\n\n" result += "This software is built and contains the following software:\n\n" result += "(automatically generated via [cargo-license](https://crates.io/crates/cargo-license))\n\n" for item in data: license = item["license"] name = item["name"] version = item["version"] repository = item["repository"] result += "------------------\n\n" result += f"### {name} {version}\n* source: [{repository}]({repository})\n* license: {license}\n\n" with Path.open("LICENSE.txt", "w") as f: f.write(result) apache-datafusion-python-54.0.0/dev/python_lint.sh000077500000000000000000000017671521727353300222020ustar00rootroot00000000000000#!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # This script runs all the Rust lints locally the same way the # DataFusion CI does set -e source .venv/bin/activate flake8 --exclude venv,benchmarks/db-benchmark --ignore=E501,W503 black --line-length 79 . apache-datafusion-python-54.0.0/dev/release/000077500000000000000000000000001521727353300207015ustar00rootroot00000000000000apache-datafusion-python-54.0.0/dev/release/README.md000066400000000000000000000256721521727353300221740ustar00rootroot00000000000000 # DataFusion Python Release Process Development happens on the `main` branch, and most of the time, we depend on DataFusion using GitHub dependencies rather than using an official release from crates.io. This allows us to pick up new features and bug fixes frequently by creating PRs to move to a later revision of the code. It also means we can incrementally make updates that are required due to changes in DataFusion rather than having a large amount of work to do when the next official release is available. When there is a new official release of DataFusion, we update the `main` branch to point to that, update the version number, and create a new release branch, such as `branch-53`. Once this branch is created, we switch the `main` branch back to using GitHub dependencies. The release activity (such as generating the changelog) can then happen on the release branch without blocking ongoing development in the `main` branch. We can cherry-pick commits from the `main` branch into `branch-53` as needed and then create new patch releases from that branch. ## Upstream Sync Between releases the `main` branch is periodically synced to a newer upstream `apache/datafusion` version. This is broken into a three-PR workflow (bump + fix breakage, consolidate transitive deps, fill API and documentation gaps). See [`upstream-sync.md`](upstream-sync.md) for the full process. ## Detailed Guide ### Pre-requisites Releases can currently only be created by PMC members due to the permissions needed. You will need a GitHub Personal Access Token. Follow [these instructions](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token) to generate one if you do not already have one. You will need a PyPI API token. Create one at https://test.pypi.org/manage/account/#api-tokens, setting the “Scope” to “Entire account”. You will also need access to the [datafusion](https://test.pypi.org/project/datafusion/) project on testpypi. ### Preparing the `main` Branch Before creating a new release: - We need to ensure that the main branch does not have any GitHub dependencies - Confirm the upstream sync workflow in [`upstream-sync.md`](upstream-sync.md) has been completed for this release cycle (crate bump + breakage fixes, transitive dependency consolidation, and the `/check-upstream` and `/audit-skill-md` passes). Any gaps surfaced by those skills should land before the release branch is cut. - a PR should be created and merged to update the major version number of the project - A new release branch should be created, such as `branch-53` - It is best to push this branch to the apache repository rather than a personal fork in case patch releases are required. ## Preparing a Release Candidate ### Change Log We maintain a `CHANGELOG.md` so our users know what has been changed between releases. The changelog is generated using a Python script: ```bash $ uv sync --group release $ GITHUB_TOKEN= ./dev/release/generate-changelog.py 52.0.0 HEAD 53.0.0 > dev/changelog/53.0.0.md ``` This script creates a changelog from GitHub PRs based on the labels associated with them as well as looking for titles starting with `feat:`, `fix:`, or `docs:` . The script will produce output similar to: ``` Fetching list of commits between 52.0.0 and HEAD Fetching pull requests Categorizing pull requests Generating changelog content ``` ### Update the version number The only place you should need to update the version is in the root `Cargo.toml`. You will need to update this both in the workspace section and also in the dependencies. After updating the toml file, run `cargo update` to update the cargo lock file. If you do not want to update all the dependencies, you can instead run `cargo build` which should only update the version number for `datafusion-python`. ### Tag the Repository Commit the changes to the changelog and version. Assuming you have set up a remote to the `apache` repository rather than your personal fork, you need to push a tag to start the CI process for release candidates. The following assumes the upstream repository is called `apache`. ```bash git tag 53.0.0-rc1 git push apache 53.0.0-rc1 ``` ### Create a source release ```bash ./dev/release/create-tarball.sh 53.0.0 1 ``` This will also create the email template to send to the mailing list. Create a draft email using this content, but do not send until after completing the next step. ### Publish Python Artifacts to testpypi This section assumes some familiarity with publishing Python packages to PyPi. For more information, refer to \ [this tutorial](https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives). #### Publish Python Wheels to testpypi Pushing an `rc` tag to the release branch will cause a GitHub Workflow to run that will build the Python wheels. Go to https://github.com/apache/datafusion-python/actions and look for an action named "Python Release Build" that has run against the pushed tag. Click on the action and scroll down to the bottom of the page titled "Artifacts". Download `dist.zip`. It should contain files such as: ```text datafusion-53.0.0-cp37-abi3-macosx_10_7_x86_64.whl datafusion-53.0.0-cp37-abi3-macosx_11_0_arm64.whl datafusion-53.0.0-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl datafusion-53.0.0-cp37-abi3-win_amd64.whl ``` Upload the wheels to testpypi. ```bash unzip dist.zip python3 -m pip install --upgrade setuptools twine build python3 -m twine upload --repository testpypi datafusion-53.0.0-cp37-abi3-*.whl ``` When prompted for username, enter `__token__`. When prompted for a password, enter a valid GitHub Personal Access Token #### Publish Python Source Distribution to testpypi Download the source tarball from the Apache server created in the previous step, untar it, and run: ```bash maturin sdist ``` This will create a file named `dist/datafusion-53.0.0.tar.gz`. Upload this to testpypi: ```bash python3 -m twine upload --repository testpypi dist/datafusion-53.0.0.tar.gz ``` ### Run Verify Release Candidate Workflow Before sending the vote email, run the manually triggered GitHub Actions workflow "Verify Release Candidate" and confirm all matrix jobs pass across the OS/architecture matrix (for example, Linux, macOS, and Windows runners): 1. Go to https://github.com/apache/datafusion-python/actions/workflows/verify-release-candidate.yml 2. Click "Run workflow" 3. Set `version` to the release version (for example, `53.0.0`) 4. Set `rc_number` to the RC number (for example, `1`) 5. Wait for all jobs to complete successfully Include a short note in the vote email template that this workflow was run across all OS/architecture matrix entries and that all jobs passed. ```text Verification note: The manually triggered "Verify Release Candidate" workflow was run for version and rc_number across all configured OS/architecture matrix entries, and all matrix jobs completed successfully. ``` ### Send the Email Send the email to start the vote. ## Verifying a Release Releases may be verified using `verify-release-candidate.sh`: ```bash git clone https://github.com/apache/datafusion-python.git dev/release/verify-release-candidate.sh 53.0.0 1 ``` Alternatively, one can run unit tests against a testpypi release candidate: ```bash # clone a fresh repo git clone https://github.com/apache/datafusion-python.git cd datafusion-python # checkout the release commit git fetch --tags git checkout 53.0.0-rc1 git submodule update --init --recursive # create the env python3 -m venv .venv source .venv/bin/activate # install release candidate pip install --extra-index-url https://test.pypi.org/simple/ datafusion==53.0.0 # install test dependencies pip install pytest numpy pytest-asyncio # run the tests pytest --import-mode=importlib python/tests -vv ``` Try running one of the examples from the top-level README, or write some custom Python code to query some available data files. ## Publishing a Release ### Publishing Apache Source Release Once the vote passes, we can publish the release. Create the source release tarball: ```bash ./dev/release/release-tarball.sh 53.0.0 1 ``` ### Publishing Rust Crate to crates.io Some projects depend on the Rust crate directly, so we publish this to crates.io ```shell cargo publish --workspace ``` ### Publishing Python Artifacts to PyPi Go to the Test PyPI page of Datafusion, and download [all published artifacts](https://test.pypi.org/project/datafusion/#files) under `dist-release/` directory. Then proceed uploading them using `twine`: ```bash twine upload --repository pypi dist-release/* ``` ### Publish Python Artifacts to conda-forge Pypi packages auto upload to conda-forge via [datafusion feedstock](https://github.com/conda-forge/datafusion-feedstock) ### Push the Release Tag ```bash git checkout 53.0.0-rc1 git tag 53.0.0 git push apache 53.0.0 ``` ### Add the release to Apache Reporter Add the release to https://reporter.apache.org/addrelease.html?datafusion with a version name prefixed with `DATAFUSION-PYTHON`, for example `DATAFUSION-PYTHON-53.0.0`. The release information is used to generate a template for a board report (see example from Apache Arrow [here](https://github.com/apache/arrow/pull/14357)). ### Delete old RCs and Releases See the ASF documentation on [when to archive](https://www.apache.org/legal/release-policy.html#when-to-archive) for more information. #### Deleting old release candidates from `dev` svn Release candidates should be deleted once the release is published. Get a list of DataFusion release candidates: ```bash svn ls https://dist.apache.org/repos/dist/dev/datafusion | grep datafusion-python ``` Delete a release candidate: ```bash svn delete -m "delete old DataFusion RC" https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-53.0.0-rc1/ ``` #### Deleting old releases from `release` svn Only the latest release should be available. Delete old releases after publishing the new release. Get a list of DataFusion releases: ```bash svn ls https://dist.apache.org/repos/dist/release/datafusion | grep datafusion-python ``` Delete a release: ```bash svn delete -m "delete old DataFusion release" https://dist.apache.org/repos/dist/release/datafusion/datafusion-python-52.0.0 ``` apache-datafusion-python-54.0.0/dev/release/check-rat-report.py000066400000000000000000000037261521727353300244350ustar00rootroot00000000000000#!/usr/bin/python ############################################################################## # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. ############################################################################## import fnmatch import re import sys import xml.etree.ElementTree as ET from pathlib import Path if len(sys.argv) != 3: sys.stderr.write("Usage: %s exclude_globs.lst rat_report.xml\n" % sys.argv[0]) sys.exit(1) exclude_globs_filename = sys.argv[1] xml_filename = sys.argv[2] globs = [line.strip() for line in Path.open(exclude_globs_filename)] tree = ET.parse(xml_filename) root = tree.getroot() resources = root.findall("resource") all_ok = True for r in resources: approvals = r.findall("license-approval") if not approvals or approvals[0].attrib["name"] == "true": continue clean_name = re.sub("^[^/]+/", "", r.attrib["name"]) excluded = False for g in globs: if fnmatch.fnmatch(clean_name, g): excluded = True break if not excluded: sys.stdout.write( "NOT APPROVED: %s (%s): %s\n" % (clean_name, r.attrib["name"], approvals[0].attrib["name"]) ) all_ok = False if not all_ok: sys.exit(1) print("OK") sys.exit(0) apache-datafusion-python-54.0.0/dev/release/create-tarball.sh000077500000000000000000000112401521727353300241200ustar00rootroot00000000000000#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # # Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/create-tarball.sh # This script creates a signed tarball in # dev/dist/apache-datafusion-python--.tar.gz and uploads it to # the "dev" area of the dist.apache.arrow repository and prepares an # email for sending to the dev@datafusion.apache.org list for a formal # vote. # # See release/README.md for full release instructions # # Requirements: # # 1. gpg setup for signing and have uploaded your public # signature to https://pgp.mit.edu/ # # 2. Logged into the apache svn server with the appropriate # credentials # # 3. Install the requests python package # # # Based in part on 02-source.sh from apache/arrow # set -e SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" SOURCE_TOP_DIR="$(cd "${SOURCE_DIR}/../../" && pwd)" if [ "$#" -ne 2 ]; then echo "Usage: $0 " echo "ex. $0 4.1.0 2" exit fi if [[ -z "${GH_TOKEN}" ]]; then echo "Please set personal github token through GH_TOKEN environment variable" exit fi version=$1 rc=$2 tag="${version}-rc${rc}" echo "Attempting to create ${tarball} from tag ${tag}" release_hash=$(cd "${SOURCE_TOP_DIR}" && git rev-list --max-count=1 ${tag}) release=apache-datafusion-python-${version} distdir=${SOURCE_TOP_DIR}/dev/dist/${release}-rc${rc} tarname=${release}.tar.gz tarball=${distdir}/${tarname} url="https://dist.apache.org/repos/dist/dev/datafusion/${release}-rc${rc}" if [ -z "$release_hash" ]; then echo "Cannot continue: unknown git tag: ${tag}" fi echo "Draft email for dev@datafusion.apache.org mailing list" echo "" echo "---------------------------------------------------------" cat < containing the files in git at $release_hash # the files in the tarball are prefixed with {version} (e.g. 4.0.1) mkdir -p ${distdir} (cd "${SOURCE_TOP_DIR}" && git archive ${release_hash} --prefix ${release}/ | gzip > ${tarball}) echo "Running rat license checker on ${tarball}" ${SOURCE_DIR}/run-rat.sh ${tarball} echo "Signing tarball and creating checksums" gpg --armor --output ${tarball}.asc --detach-sig ${tarball} # create signing with relative path of tarball # so that they can be verified with a command such as # shasum --check apache-datafusion-python-4.1.0-rc2.tar.gz.sha512 (cd ${distdir} && shasum -a 256 ${tarname}) > ${tarball}.sha256 (cd ${distdir} && shasum -a 512 ${tarname}) > ${tarball}.sha512 echo "Uploading to datafusion dist/dev to ${url}" svn co --depth=empty https://dist.apache.org/repos/dist/dev/datafusion ${SOURCE_TOP_DIR}/dev/dist svn add ${distdir} svn ci -m "Apache DataFusion Python ${version} ${rc}" ${distdir} apache-datafusion-python-54.0.0/dev/release/generate-changelog.py000077500000000000000000000145021521727353300247770ustar00rootroot00000000000000#!/usr/bin/env python # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. import argparse import os import re import subprocess import sys from github import Github def print_pulls(repo_name, title, pulls) -> None: if len(pulls) > 0: print(f"**{title}:**") print() for pull, commit in pulls: url = f"https://github.com/{repo_name}/pull/{pull.number}" print(f"- {pull.title} [#{pull.number}]({url}) ({commit.author.login})") print() def generate_changelog(repo, repo_name, tag1, tag2, version) -> None: # get a list of commits between two tags print(f"Fetching list of commits between {tag1} and {tag2}", file=sys.stderr) comparison = repo.compare(tag1, tag2) # get the pull requests for these commits print("Fetching pull requests", file=sys.stderr) unique_pulls = [] all_pulls = [] for commit in comparison.commits: pulls = commit.get_pulls() for pull in pulls: # there can be multiple commits per PR if squash merge is not being used and # in this case we should get all the author names, but for now just pick one if pull.number not in unique_pulls: unique_pulls.append(pull.number) all_pulls.append((pull, commit)) # we split the pulls into categories breaking = [] bugs = [] docs = [] enhancements = [] performance = [] other = [] # categorize the pull requests based on GitHub labels print("Categorizing pull requests", file=sys.stderr) for pull, commit in all_pulls: # see if PR title uses Conventional Commits cc_type = "" cc_breaking = "" parts = re.findall(r"^([a-z]+)(\([a-z]+\))?(!)?:", pull.title) if len(parts) == 1: parts_tuple = parts[0] cc_type = parts_tuple[0] # fix, feat, docs, chore # cc_scope = parts_tuple[1] # component within project cc_breaking = parts_tuple[2] == "!" labels = [label.name for label in pull.labels] if "api change" in labels or cc_breaking: breaking.append((pull, commit)) elif "bug" in labels or cc_type == "fix": bugs.append((pull, commit)) elif "performance" in labels or cc_type == "perf": performance.append((pull, commit)) elif "enhancement" in labels or cc_type == "feat": enhancements.append((pull, commit)) elif "documentation" in labels or cc_type == "docs" or cc_type == "doc": docs.append((pull, commit)) else: other.append((pull, commit)) # produce the changelog content print("Generating changelog content", file=sys.stderr) # ASF header print("""\n""") print(f"# Apache DataFusion Python {version} Changelog\n") # get the number of commits commit_count = subprocess.check_output( f"git log --pretty=oneline {tag1}..{tag2} | wc -l", shell=True, text=True ).strip() # get number of contributors contributor_count = subprocess.check_output( f"git shortlog -sn {tag1}..{tag2} | wc -l", shell=True, text=True ).strip() print( f"This release consists of {commit_count} commits from {contributor_count} contributors. " f"See credits at the end of this changelog for more information.\n" ) print_pulls(repo_name, "Breaking changes", breaking) print_pulls(repo_name, "Performance related", performance) print_pulls(repo_name, "Implemented enhancements", enhancements) print_pulls(repo_name, "Fixed bugs", bugs) print_pulls(repo_name, "Documentation updates", docs) print_pulls(repo_name, "Other", other) # show code contributions credits = subprocess.check_output( f"git shortlog -sn {tag1}..{tag2}", shell=True, text=True ).rstrip() print("## Credits\n") print( "Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) " "per contributor.\n" ) print("```") print(credits) print("```\n") print( "Thank you also to everyone who contributed in other ways such as filing issues, reviewing " "PRs, and providing feedback on this release.\n" ) def cli(args=None) -> None: """Process command line arguments.""" if not args: args = sys.argv[1:] parser = argparse.ArgumentParser() parser.add_argument("tag1", help="The previous commit or tag (e.g. 0.1.0)") parser.add_argument("tag2", help="The current commit or tag (e.g. HEAD)") parser.add_argument( "version", help="The version number to include in the changelog" ) args = parser.parse_args() token = os.getenv("GITHUB_TOKEN") project = "apache/datafusion-python" g = Github(token) repo = g.get_repo(project) generate_changelog(repo, project, args.tag1, args.tag2, args.version) if __name__ == "__main__": cli() apache-datafusion-python-54.0.0/dev/release/rat_exclude_files.txt000066400000000000000000000013321521727353300251220ustar00rootroot00000000000000*.npmrc *.gitignore *.dockerignore .gitmodules *_generated.h *_generated.js *_generated.ts *.csv *.json *.snap .github/ISSUE_TEMPLATE/*.md .github/pull_request_template.md CHANGELOG.md dev/release/rat_exclude_files.txt MANIFEST.in __init__.pxd __init__.py *.html *.sgml *.css *.png *.ico *.svg *.devhelp2 *.scss .gitattributes requirements.txt *requirements*.txt **/testdata/* ci/* **/*.svg **/*.csv **/*.json **/*.sql venv/* parquet/* testing/* target/* **/target/* Cargo.lock **/Cargo.lock .history *rat.txt */.git .github/* benchmarks/tpch/queries/q*.sql benchmarks/tpch/create_tables.sql .cargo/config.toml **/.cargo/config.toml uv.lock examples/tpch/answers_sf1/*.tbl **/SKILL.md docs/source/llms.txt CLAUDE.md .claude/skillsapache-datafusion-python-54.0.0/dev/release/release-tarball.sh000077500000000000000000000046261521727353300243070ustar00rootroot00000000000000#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # # Adapted from https://github.com/apache/arrow-rs/tree/master/dev/release/release-tarball.sh # This script copies a tarball from the "dev" area of the # dist.apache.arrow repository to the "release" area # # This script should only be run after the release has been approved # by the arrow PMC committee. # # See release/README.md for full release instructions # # Based in part on post-01-upload.sh from apache/arrow set -e set -u if [ "$#" -ne 2 ]; then echo "Usage: $0 " echo "ex. $0 4.1.0 2" exit fi version=$1 rc=$2 read -r -p "Proceed to release tarball for ${version}-rc${rc}? [y/N]: " answer answer=${answer:-no} if [ "${answer}" != "y" ]; then echo "Cancelled tarball release!" exit 1 fi tmp_dir=tmp-apache-datafusion-python-dist echo "Recreate temporary directory: ${tmp_dir}" rm -rf ${tmp_dir} mkdir -p ${tmp_dir} echo "Clone dev dist repository" svn \ co \ https://dist.apache.org/repos/dist/dev/datafusion/apache-datafusion-python-${version}-rc${rc} \ ${tmp_dir}/dev echo "Clone release dist repository" svn co https://dist.apache.org/repos/dist/release/datafusion ${tmp_dir}/release echo "Copy ${version}-rc${rc} to release working copy" release_version=datafusion-python-${version} mkdir -p ${tmp_dir}/release/${release_version} cp -r ${tmp_dir}/dev/* ${tmp_dir}/release/${release_version}/ svn add ${tmp_dir}/release/${release_version} echo "Commit release" svn ci -m "Apache DataFusion Python ${version}" ${tmp_dir}/release echo "Clean up" rm -rf ${tmp_dir} echo "Success! The release is available here:" echo " https://dist.apache.org/repos/dist/release/datafusion/${release_version}" apache-datafusion-python-54.0.0/dev/release/run-rat.sh000077500000000000000000000027721521727353300226400ustar00rootroot00000000000000#!/bin/bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # RAT_VERSION=0.13 # download apache rat if [ ! -f apache-rat-${RAT_VERSION}.jar ]; then curl -s https://repo1.maven.org/maven2/org/apache/rat/apache-rat/${RAT_VERSION}/apache-rat-${RAT_VERSION}.jar > apache-rat-${RAT_VERSION}.jar fi RAT="java -jar apache-rat-${RAT_VERSION}.jar -x " RELEASE_DIR=$(cd "$(dirname "$BASH_SOURCE")"; pwd) # generate the rat report $RAT $1 > rat.txt python $RELEASE_DIR/check-rat-report.py $RELEASE_DIR/rat_exclude_files.txt rat.txt > filtered_rat.txt cat filtered_rat.txt UNAPPROVED=`cat filtered_rat.txt | grep "NOT APPROVED" | wc -l` if [ "0" -eq "${UNAPPROVED}" ]; then echo "No unapproved licenses" else echo "${UNAPPROVED} unapproved licences. Check rat report: rat.txt" exit 1 fi apache-datafusion-python-54.0.0/dev/release/upstream-sync.md000066400000000000000000000174541521727353300240500ustar00rootroot00000000000000 # Upstream Sync Process This document describes how to sync `datafusion-python` to a new version of the upstream `apache/datafusion` Rust crates. This is a recurring task: between official releases the `main` branch tracks DataFusion via crates.io or GitHub dependencies, and we periodically bump those dependencies to pick up new features and bug fixes. The work is broken into **three sequential PRs** rather than landing as one large change. Splitting reviews along these lines keeps each PR focused, makes breakage easier to bisect, and lets reviewers concentrate on one concern at a time. ## PR 1: Bump DataFusion crate dependencies and fix breakage **Goal:** update the upstream `datafusion` crate version and make the project build, test, and lint cleanly against it. 1. In the root `Cargo.toml`, update: - `[workspace.package].version` to the new major (the `datafusion-python` major tracks the upstream `datafusion` major, so a 53→54 bump moves this from `53.0.0` to `54.0.0`), and - every `datafusion` / `datafusion-*` entry in `[workspace.dependencies]` to the same new major. Per-crate manifests under `crates/` inherit these pins via `workspace = true` and need no edit. 2. Update `Cargo.lock` for the datafusion family only — leave unrelated transitives at their current pins so PR 2 can address them deliberately. List every `datafusion-*` workspace dependency with `-p`: ```bash cargo update \ -p datafusion \ -p datafusion-substrait \ -p datafusion-proto \ -p datafusion-ffi \ -p datafusion-catalog \ -p datafusion-common \ -p datafusion-functions-aggregate \ -p datafusion-functions-window \ -p datafusion-expr ``` Or pin exact versions with `--precise`, one crate at a time: ```bash cargo update -p datafusion --precise 54.0.0 # repeat for each datafusion-* sibling ``` A bare `cargo update` would refresh every transitive crate and blur the diff between PR 1 and PR 2. 3. Run the standard build and test commands and address compilation errors, API renames, signature changes, and behavior changes: - `cargo build` - `cargo test` - `pytest` - `pre-commit run --all-files` 4. Fix only what's needed to restore green CI. Resist the urge to bundle unrelated cleanups — those belong in their own PR. 5. If a breaking change in upstream requires a user-facing API change in `datafusion-python`, add the `api change` label and document the change in the PR description so it surfaces in the changelog. **Reference PRs:** [#1311](https://github.com/apache/datafusion-python/pull/1311) (DF51), [#1337](https://github.com/apache/datafusion-python/pull/1337) (DF52). ## PR 2: Consolidate transitive dependencies **Goal:** after the upstream bump, the dependency tree may have multiple versions of the same transitive crate (for example, two `arrow` versions, two `object_store` versions). Reconcile these so we ship a single coherent set. 1. Inspect the lockfile for duplicates: ```bash cargo tree --duplicates ``` 2. For each duplicate that matters (Arrow, `object_store`, `parquet`, `tokio`, `arrow-flight`, etc.), update our direct dependency declarations in `Cargo.toml` to versions compatible with what upstream DataFusion now pulls in. The goal is one version of each ecosystem-critical crate. 3. Re-run `cargo update` and re-run the full test matrix. Some duplicates are benign (small leaf crates with no FFI surface) and can be left alone if reconciliation would force a much larger change. Use judgment. 4. If consolidating forces a behavioral change visible to users (for example, a newer `pyarrow`-compatible Arrow version), call it out in the PR description. Keeping this work separate from PR 1 means PR 1 stays a "make it compile" review and PR 2 stays a "tidy the dependency graph" review. ## PR 3: Fill API and documentation gaps **Goal:** with the upstream version locked in, identify new APIs that landed upstream and decide whether to expose them, and update agent-facing documentation so it still matches the surface we ship. 1. Run the `check-upstream` skill (`.ai/skills/check-upstream/SKILL.md`) to diff the upstream Rust API against what's exposed in `python/datafusion/`. The skill covers scalar/aggregate/window/table functions, `DataFrame` methods, `SessionContext` methods, and FFI types. Invoke it from the assistant with `/check-upstream` (optionally scoped to one area, e.g. `/check-upstream scalar functions`). 2. For each gap, decide whether to: - Expose it now (small, obvious additions can land in this PR). - File a tracking issue (anything non-trivial — separate PR per feature keeps reviews focused). - Skip it (internal-only or already covered by an existing API; record the decision in the "Evaluated and not requiring exposure" sections of the skill so future runs don't re-flag it). 3. (Optional) Run the `make-pythonic` skill (`.ai/skills/make-pythonic/SKILL.md`) over any newly exposed APIs to align signatures with the project's Pythonic style (accepting plain strings for column names, raw Python values where auto-wrapping applies, etc.). Invoke it from the assistant with `/make-pythonic`. Running this *before* the audit step means examples in `SKILL.md` get updated to the final signature in one pass instead of churning twice. Larger reshapes still belong in their own PR. 4. Run the `audit-skill-md` skill (`.ai/skills/audit-skill-md/SKILL.md`) to cross-reference the user-facing skill at [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) against the current public API. The skill flags stale function names, missing newly exposed APIs, examples that drifted from idiomatic style, and missing version notes. Invoke it from the assistant with `/audit-skill-md` (optionally scoped, e.g. `/audit-skill-md dataframe`). Apply the resulting edits to `SKILL.md` and to the relevant RST pages under `docs/source/user-guide/common-operations/`. 5. If new aggregate or window functions were exposed in step 2, also update: - `docs/source/user-guide/common-operations/aggregations.rst` - `docs/source/user-guide/common-operations/windows.rst` ## Why three PRs - **Bisectable.** If a regression appears, `git bisect` lands on the responsible PR (compile fix, dependency consolidation, or API addition) rather than a single mega-commit. - **Reviewable.** Each PR has a single concern. Reviewers reading PR 1 don't need to also reason about whether new APIs are well-named. - **Skippable.** Some upstream syncs are pure version bumps with no new APIs worth exposing. PR 3 can be empty or merged as a no-op if the audit comes back clean. ## Related documents - [`README.md`](README.md) — the broader release process (this sync work feeds into the next official release). - [`.ai/skills/check-upstream/SKILL.md`](../../.ai/skills/check-upstream/SKILL.md) — API coverage audit. - [`skills/datafusion_python/SKILL.md`](../../skills/datafusion_python/SKILL.md) — user-facing agent guide kept in sync via PR 3. apache-datafusion-python-54.0.0/dev/release/verify-release-candidate.sh000077500000000000000000000115531521727353300261010ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # case $# in 2) VERSION="$1" RC_NUMBER="$2" ;; *) echo "Usage: $0 X.Y.Z RC_NUMBER" exit 1 ;; esac set -e set -x set -o pipefail SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]:-$0}")" && pwd)" DATAFUSION_PYTHON_DIR="$(dirname $(dirname ${SOURCE_DIR}))" DATAFUSION_PYTHON_DIST_URL='https://dist.apache.org/repos/dist/dev/datafusion' download_dist_file() { curl \ --silent \ --show-error \ --fail \ --location \ --remote-name $DATAFUSION_PYTHON_DIST_URL/$1 } download_rc_file() { download_dist_file apache-datafusion-python-${VERSION}-rc${RC_NUMBER}/$1 } import_gpg_keys() { download_dist_file KEYS gpg --import KEYS } if type shasum >/dev/null 2>&1; then sha256_verify="shasum -a 256 -c" sha512_verify="shasum -a 512 -c" else sha256_verify="sha256sum -c" sha512_verify="sha512sum -c" fi fetch_archive() { local dist_name=$1 download_rc_file ${dist_name}.tar.gz download_rc_file ${dist_name}.tar.gz.asc download_rc_file ${dist_name}.tar.gz.sha256 download_rc_file ${dist_name}.tar.gz.sha512 verify_dir_artifact_signatures } verify_dir_artifact_signatures() { # verify the signature and the checksums of each artifact find . -name '*.asc' | while read sigfile; do artifact=${sigfile/.asc/} gpg --verify $sigfile $artifact || exit 1 # go into the directory because the checksum files contain only the # basename of the artifact pushd $(dirname $artifact) base_artifact=$(basename $artifact) ${sha256_verify} $base_artifact.sha256 || exit 1 ${sha512_verify} $base_artifact.sha512 || exit 1 popd done } setup_tempdir() { cleanup() { if [ "${TEST_SUCCESS}" = "yes" ]; then rm -fr "${DATAFUSION_PYTHON_TMPDIR}" else echo "Failed to verify release candidate. See ${DATAFUSION_PYTHON_TMPDIR} for details." fi } if [ -z "${DATAFUSION_PYTHON_TMPDIR}" ]; then # clean up automatically if DATAFUSION_PYTHON_TMPDIR is not defined DATAFUSION_PYTHON_TMPDIR=$(mktemp -d -t "$1.XXXXX") trap cleanup EXIT else # don't clean up automatically mkdir -p "${DATAFUSION_PYTHON_TMPDIR}" fi } test_source_distribution() { # install rust toolchain export RUSTUP_HOME=$PWD/test-rustup export CARGO_HOME=$PWD/test-rustup curl https://sh.rustup.rs -sSf | sh -s -- -y --no-modify-path # On Unix, rustup creates an env file. On Windows GitHub runners (MSYS bash), # that file may not exist, so fall back to adding Cargo bin directly. if [ -f "$CARGO_HOME/env" ]; then # shellcheck disable=SC1090 source "$CARGO_HOME/env" elif [ -f "$RUSTUP_HOME/env" ]; then # shellcheck disable=SC1090 source "$RUSTUP_HOME/env" else export PATH="$CARGO_HOME/bin:$PATH" fi # build and test rust # raises on any formatting errors rustup component add rustfmt --toolchain stable cargo fmt --all -- --check # Clone testing repositories into the expected location git clone https://github.com/apache/arrow-testing.git testing git clone https://github.com/apache/parquet-testing.git parquet-testing python3 -m venv .venv if [ -x ".venv/bin/python" ]; then VENV_PYTHON=".venv/bin/python" elif [ -x ".venv/Scripts/python.exe" ]; then VENV_PYTHON=".venv/Scripts/python.exe" elif [ -x ".venv/Scripts/python" ]; then VENV_PYTHON=".venv/Scripts/python" else echo "Unable to find python executable in virtual environment" exit 1 fi "$VENV_PYTHON" -m pip install -U pip "$VENV_PYTHON" -m pip install -U maturin "$VENV_PYTHON" -m maturin develop #TODO: we should really run tests here as well #python3 -m pytest if ( find -iname 'Cargo.toml' | xargs grep SNAPSHOT ); then echo "Cargo.toml version should not contain SNAPSHOT for releases" exit 1 fi } TEST_SUCCESS=no setup_tempdir "datafusion-python-${VERSION}" echo "Working in sandbox ${DATAFUSION_PYTHON_TMPDIR}" cd ${DATAFUSION_PYTHON_TMPDIR} dist_name="apache-datafusion-python-${VERSION}" import_gpg_keys fetch_archive ${dist_name} tar xf ${dist_name}.tar.gz pushd ${dist_name} test_source_distribution popd TEST_SUCCESS=yes echo 'Release candidate looks good!' exit 0 apache-datafusion-python-54.0.0/dev/rust_lint.sh000077500000000000000000000021251521727353300216430ustar00rootroot00000000000000#!/usr/bin/env bash # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # This script runs all the Rust lints locally the same way the # DataFusion CI does set -e if ! command -v cargo-tomlfmt &> /dev/null; then echo "Installing cargo-tomlfmt using cargo" cargo install cargo-tomlfmt fi ci/scripts/rust_fmt.sh ci/scripts/rust_clippy.sh ci/scripts/rust_toml_fmt.sh apache-datafusion-python-54.0.0/docs/000077500000000000000000000000001521727353300174335ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/.gitignore000066400000000000000000000001061521727353300214200ustar00rootroot00000000000000pokemon.csv yellow_trip_data.parquet yellow_tripdata_2021-01.parquet apache-datafusion-python-54.0.0/docs/Makefile000066400000000000000000000026431521727353300211000ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # # Minimal makefile for Sphinx documentation # # You can set these variables from the command line, and also # from the environment for the first two. SPHINXOPTS ?= SPHINXBUILD ?= sphinx-build SOURCEDIR = source BUILDDIR = build # Put it first so that "make" without argument is like "make help". help: @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) .PHONY: help Makefile # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) --fail-on-warningapache-datafusion-python-54.0.0/docs/README.md000066400000000000000000000046171521727353300207220ustar00rootroot00000000000000 # DataFusion Documentation This folder contains the source content of the [Python API](./source/api). This is published to https://datafusion.apache.org/python by a GitHub action when changes are merged to the main branch. ## Dependencies It's recommended to install build dependencies and build the documentation inside a Python `venv` using `uv`. To prepare building the documentation run the following on the root level of the project: ```bash # Set up a virtual environment with the documentation dependencies uv sync --dev --group docs --no-install-package datafusion ``` ## Build & Preview Run the provided script to build the HTML pages. ```bash # Build the repository uv run --no-project maturin develop --uv # Build the documentation uv run --no-project docs/build.sh ``` The HTML will be generated into a `build` directory in `docs`. Preview the site on Linux by running this command. ```bash firefox docs/build/html/index.html ``` ## Release Process This documentation is hosted at https://datafusion.apache.org/python When the PR is merged to the `main` branch of the DataFusion repository, a [github workflow](https://github.com/apache/datafusion-python/blob/main/.github/workflows/build.yml) which: 1. Builds the html content 2. Pushes the html content to the [`asf-site`](https://github.com/apache/datafusion-python/tree/asf-site) branch in this repository. The Apache Software Foundation provides https://arrow.apache.org/, which serves content based on the configuration in [.asf.yaml](https://github.com/apache/datafusion-python/blob/main/.asf.yaml), which specifies the target as https://datafusion.apache.org/python. apache-datafusion-python-54.0.0/docs/build.sh000077500000000000000000000041551521727353300210760ustar00rootroot00000000000000#!/usr/bin/env bash # # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # set -e original_dir=$(pwd) script_dir=$(dirname "$(realpath "$0")") cd "$script_dir" || exit if [ ! -f pokemon.csv ]; then curl -O https://gist.githubusercontent.com/ritchie46/cac6b337ea52281aa23c049250a4ff03/raw/89a957ff3919d90e6ef2d34235e6bf22304f3366/pokemon.csv fi if [ ! -f yellow_tripdata_2021-01.parquet ]; then curl -O https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet fi rm -rf build 2> /dev/null rm -rf temp 2> /dev/null mkdir temp cp -rf source/* temp/ # myst-nb executes each page as a notebook from the directory that page # lives in, so the example data files must sit alongside every page that # loads them by relative name (e.g. `ctx.read_csv("pokemon.csv")`). Symlink # them into each directory that has such a page rather than copying the # 20 MB parquet repeatedly. for d in temp temp/user-guide temp/user-guide/common-operations; do ln -sf "$script_dir/pokemon.csv" "$d/pokemon.csv" ln -sf "$script_dir/yellow_tripdata_2021-01.parquet" "$d/yellow_tripdata_2021-01.parquet" done # myst-nb runs `{code-cell}` blocks against a Jupyter kernel named "python3". # Register the active environment's interpreter as that kernel (idempotent). python -m ipykernel install --sys-prefix --name python3 --display-name "Python 3" make SOURCEDIR=`pwd`/temp html cd "$original_dir" || exit apache-datafusion-python-54.0.0/docs/make.bat000066400000000000000000000030751521727353300210450ustar00rootroot00000000000000@rem Licensed to the Apache Software Foundation (ASF) under one @rem or more contributor license agreements. See the NOTICE file @rem distributed with this work for additional information @rem regarding copyright ownership. The ASF licenses this file @rem to you under the Apache License, Version 2.0 (the @rem "License"); you may not use this file except in compliance @rem with the License. You may obtain a copy of the License at @rem @rem http://www.apache.org/licenses/LICENSE-2.0 @rem @rem Unless required by applicable law or agreed to in writing, @rem software distributed under the License is distributed on an @rem "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY @rem KIND, either express or implied. See the License for the @rem specific language governing permissions and limitations @rem under the License. @ECHO OFF pushd %~dp0 REM Command file for Sphinx documentation if "%SPHINXBUILD%" == "" ( set SPHINXBUILD=sphinx-build ) set SOURCEDIR=source set BUILDDIR=build if "%1" == "" goto help %SPHINXBUILD% >NUL 2>NUL if errorlevel 9009 ( echo. echo.The 'sphinx-build' command was not found. Make sure you have Sphinx echo.installed, then set the SPHINXBUILD environment variable to point echo.to the full path of the 'sphinx-build' executable. Alternatively you echo.may add the Sphinx directory to PATH. echo. echo.If you don't have Sphinx installed, grab it from echo.http://sphinx-doc.org/ exit /b 1 ) %SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% goto end :help %SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% :end popdapache-datafusion-python-54.0.0/docs/mdbook/000077500000000000000000000000001521727353300207065ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/mdbook/README.md000066400000000000000000000024631521727353300221720ustar00rootroot00000000000000 # DataFusion Book This folder builds a DataFusion user guide using [mdBook](https://github.com/rust-lang/mdBook). ## Build and run book locally Build the latest files with `mdbook build`. Open the book locally by running `open book/index.html`. ## Install mdBook Download the `mdbook` binary or run `cargo install mdbook`. Then manually open it, so you have permissions to run it on your Mac. Add it to your path with a command like this so you can easily run the commands: `mv ~/Downloads/mdbook /Users/matthew.powers/.local/bin`. apache-datafusion-python-54.0.0/docs/mdbook/book.toml000066400000000000000000000016261521727353300225420ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. [book] authors = ["Apache Arrow "] language = "en" multilingual = false src = "src" title = "DataFusion Book" apache-datafusion-python-54.0.0/docs/mdbook/src/000077500000000000000000000000001521727353300214755ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/mdbook/src/SUMMARY.md000066400000000000000000000020511521727353300231520ustar00rootroot00000000000000 # Summary - [Index](./index.md) - [Installation](./installation.md) - [Quickstart](./quickstart.md) - [Usage](./usage/index.md) - [Create a table](./usage/create-table.md) - [Query a table](./usage/query-table.md) - [Viewing Query Plans](./usage/query-plans.md)apache-datafusion-python-54.0.0/docs/mdbook/src/images/000077500000000000000000000000001521727353300227425ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/mdbook/src/images/datafusion-jupyterlab.png000066400000000000000000002501771521727353300300000ustar00rootroot00000000000000PNG  IHDRzD @iCCPICC ProfileHWXS[@B %&RBhFHb E.";bgQl ʺX+oR@}{}sϙsg)Xj+*Ć0&0H]/fEGGXۿw"k;ȴ_&_8˅x%O,)(ͧeV-BP3Rx&> q3*j\$U3 yP/΀777q*6F LN4ӆ4܌! 9ig:w͑U-S+3pVW|=(%SG yl3 !%. @ W:UXX⅂8fI^Z.aDW4;)(1ZQf|-  vώ Wڌ.dGH- BXa$8Vi_?8_ls 2Cy\yp.U0#18 0H1w[ JS|*qN7x3] c x :^'^ Vă/ HaMy [{{ᝢ'pdpP2#="xEO h\W !uU\@P>"<8x/ yKO!#w.<od"t#C}ВD ${3=ݞFxLI ܙ$,H1tpq]8 V- '#;Q0?4;ې,?GkPC=?g}>lbi"v  $ր`exhu=Aox,N5N=N_}w4`牧I "qEz#n -߹ysr``w.$=?aO*B.P;Ms` ܁7A Dx &3:)` J@XV` v=ip\WMpNw3 $}Dg"AH$#H"B dR@#[jd?r9\Dڐ;#y|B1T FP+t$DYh8N@3h:]EhzD;h?0UL30&Ƣ,`RjFcX/'t;'<|2> _wux3~T!E2S%rvaY: D.њb218xF|B'H${)%JHHI'IH**&***)*"br]*'Tt|&k-^(2xJe.e-rrFUULS5FU:Gu> T?i٩ƫIՖP;vG JSS%jCHٴ Z:YR>QH\^ [1KBFF&]sffb]5HZVZAZ|Z[h=cts:Σϣowj9Ye{[tt\uuTtt99Ku4hk`آaî {7\_OWW'}~~rzAg zk^:𻆨atí-FF!FbuFgzuW01MV4ya9fFitig3kbf)LtUM}&c,fXXܵ$[2-3-X|oemdުZϚc]d]c}߆jg3٦-іimjfiWawwooA9B4jDˡСᑣcccˑ#SF.y~7'7mNFi U*-(0%qSES[M[4(t3x45s,dVڬ2g\칿;(~;/i^|s?%䗚Z}M …\[_z̩bKuKҗ.u_qqh٭~w\Q1+V1VzzPHtX۰bݲu_gYPҰrQ 6odlӧͷl*JZٶmcV`{;D;:vleki Z#=~={jj[{?u2y Ͻ%jY塿jJj7ovu}]K?O]|!}Ykor\ W+zd|F8 8OXqFwj{L/i`6x P:xV+ec妁Sg2UWs/e|@EqeXIfMM*>F(iNxȠASCIIScreenshotp ̂ pHYs%%IR$iTXtXML:com.adobe.xmp 694 1480 Screenshot iDOT[([[0h @IDATxE%@s YTsP19'sBE0`€b"b@E0 jzggfggvgߺLO_Ts*Q@@@@@g ׳+"    8     Ry4     r@@@@@z)@^^vN@@@@@9     @ @^//;'     @{@@@@@^  F@@@@ @=     P/I#     @@@@@e@@@@@ 8q"     IuQi@@@@@ $@޼yփ@@@@@(rs0@@@@@"@\      PTE`     Eym@@@@@ȋ@@@@@jr%     @Q!    JP@@@@@  /*7C@@@@-k˕     E @^Tn     P[ז+A=@@@@@*@ @@@@@ -Wz PMF0uvK4^ʶiK;    JV5gѠAC[xEJRWk?Ν@1c{{dE߳#:n&K۽G<    (i] 1{ ]tQ K.K96mr=gC@@Mdgܵ'DJ  Ϭ{ R3 avan\Vh֬5o޼P׵[N # iM5Vo={_roM￟ԥߪϑGi_|EڥS>h;RkDmےE]{v   @ ,@ȸ슧Ky6]x1;jܼg-Dj9)@ݪ+ /p2L-\_of .暵Ҭf?Ix\%>6~xW`opR>n{gEVWåk;<[uUaÆڋ/z {c[@@@v,@~W㞡k۝/_A>wqGꫯLX#PɓMiT6h#2d Ɖ +mȌ[μ (US~A\LTE9:kWu{   @}(j39뷾zžyʽ+؆vN}op3oʯ?ިިqݭBM&`-蒶 [{[f˦Մz>pxl왶F62k}5K[_6gkVQ'W~5 d-͎/+kzɮԪ{#r_xG]/ ̩`֖[nY z_=}]4irW5XdVpAt&gvu{M+:묓/Jy-1+믿nRKY֭mWm6'z*5\&:@E=7|Ԭx} 7| ;m8?SO= x[mM7um-p])-7iཀྵS*bT'@Li3{W_5v횘=n̻?&ntӖ.-_Sܲj#޸hjmM]e|-Ѹnڻ5nkϥo7{#Ϟ_jk/Ѻޔ$Syӧ/_IxNlf[b:'OwXiF'c/^߼iJ!kZI=+nx@ş) /zKqòAE;q8+iΎ9I'd}98Ws5m5joMPoil̙.߿dMSNϚVuG{V3NvZh!iJ8;_je)w=6KJuAUU`RԽ[y\H=<3Ro}*_Wz{3YZksB=C@@(@Q]/Y'4r+oy8:i6i?Weϊ \߭NLW@ɿN,"6pkm /j6J;UnNHu묳VQPSNI`^W6,z]?ۢzEظyR\uUJ+ݴgSt**vޕvsgبQMA+*fd0>A:g͚eW^y~quD/䒴F>.28p`콣}5c:@&K۽G 鿺Yڢ 6E*ҳǏ&vΗؖkn8 [R<,VгAψhm#'WlWaï{dliw<v+-+-rH7{A z(a/mtS?Mvu/jhowqCcvFgJ*i_~yϸk֎:꨸]СC]p(pgʘ1c{ۅӥ{uT~I@?Oz.nr-g#FO 뮲zB5r Qw#A@@u>@I1 +нϐmI^rhպW9 ]/ VUeTOlB 9riPMV:S'L?pw.{OgVЫkS;ګ; RڮW⯽Zx?-Dzy|O?8l0oۻ^{x| HEA.wz"5 :'/; *Уސ-Zp=Te}gmڴ_g \y>p}Ti?g[ph}\-i bsk޾6ޛQHJEVfԖz݆o4TImRg%駟ezEhoM.][#~Q=k{=c}ŭ[o펭gD$6귮ߪܻa_~=PWg3KYw5ToN#g䦠2,c&L0E7+, *yРAz[J jl~y+~_|"  ^ =;~0WRYvb,n'lwn껟 tkR(?rC}g/~_}nTҥQ`|>6-p+} kYǕ6QoOK{u:zSj?t2Nu&LRk }\D@nR(hJ\ <\ k`?POhK/uͬ Jd _]OTDl=C^O?ҷo_*@^;M9cy.?_tE\}~ȯzݯ·zzꩧ/+5O_h?/ӦGM6q+DYHWQG W}?,aQ{{!:xmL;[~/_Jn|=s~ET W*)7o۟'V׃QUq?OpNv˺+Psޯ吏Jヱڏz{+TJjϜc(zAF/-(zx,=ϴo_[S:_%JJhj(73l b-5ִiS]odtJa\g}p z!ظq \U\J)3-joqWڷ[5. ~ 2UŅ~[>@@@j@Q)+e_Z*qMW2|>kmv6"`~ͻiWfkl}GvmhVm/X|^8cM.T]=ѫ[xROH6@=0  ׍VI~Np)E֫@ *קOw|+f\}[r>+ϲ/a,.ezHAoZyk|{vİS6@M]ηFi~1|ӅQǼ`a=1M1' ~u;vYwOw*PJ<%7慿k}׸ms&폿n܌~^\Eel~wڒ[ jw=S *&=H&t;\\VEnZoRଞ%gnb` bWNz[$K6V;07>v4FBVݗ J僺~'x\V6, <p,n[TJY^ms=]vG q[宻Jq3*QÌtoWB^n{n   PB: {`&9>C63RWXre퐇S5]=J=/c҆u[zk>Ůxr=Vmچ8/?˫*|TPSLq)B4 J@uNi]$ wq.5B\Wb3|-_gr=T;5m-٨ ~pԎ*&\9Unƴ4z}"@@@ clyhЬW=u7\us`>g>7lSyJ+1򭹁e>O zn*wGWڗrwO k?7tSUJ;o^YW^׸o @{O3T W=4zgT|Mns)SOE=6 x946~y W ]u4ht| {C|@/Gc_uxFJ4p[_w-F|9ն[ڸ+syTr#@E%Q`7:Xi۶_=-MHjfu⋧֪oMU|QSNʅޤI G60PO ꥬq | Ǐ믿NohpJQ9mu+ߖmp?3)(pzJ~ykmF@@u>@~kQOg?>b]p7|N,ѼJpOߧRw"L]Ϲ/ 3fLNUWJ B>211(]9T Ww}IQRd[5_)]K}xs_CA֌0@[/ ǥ@9""?ikFţ<OV JмǏ{!M wnήHK)u|J٢ABUT>RX.$/E_1 ߮}w;q~qV-@J7ߪJ sY˩A*9Sߚ޸PYX4ٔl~ onwR)ҾDsaW'@T)@uߖppLo=4UсG60r=i?G@@(@eIbgjPSA:MR})Ի[5*8 ?쳩,qѢջR%Zk ܫǩzb7h@JpBÞ \`_'ҠpDngtӺs=3n3_hy=}QBߕ'N\ r+xXz*mMb'SAS3*&>{YK6nn9:{ ۝gY䪸oj7֤ 4״ ˷T緦c*-ܒB@ǩDU_ +WZP:|:-'K)0*j r JR 5Sɵ2@@@ԋy4_q~/SS{^55|A%AM=bSjh\_)@TFm3ÀTp9S  6y䑕V+t~3@Q9ar-Sg n~p8UJA0gqR]=aQ%++1te݌h[3=Ȟ$=_=}X-K W[Z]E5@/ j`6ÁOϩS|OLr ױcG]ta/~0ʸug.9̙ckT7^ဤr矶bç}>!w<fUt9ŕWuW`GU;v-)e'3GNI}W*aYI k`鿺E ҙ\;Rpeg>/-[tƦm6hnG.y|u=7Q.g\Bȣ`hI_tIe!~v I/|.&T4riR|,zުW%tRi>Xml{{kQ {.8߬ҭ0݈+ 9n׀mv\rib{t mڴI[?ISp~~Ro(n%sO*@@@ ԋeթ nRXQ%^ >):똂R  R0FAEYV`AA)@u|Z\V0uAjZVXa/ .H7Bȵ07CqEXr^뮮שTd]V`QEנUVL7, kpM]GK&MRAni/7|.UQݕ>B%.(Wy4dܿ⍍k$@&;dLR4頹i3R3+&  m;ޭ~vdYyOt YuWCd iz{AgX^JUgSM655L05jw)PꮴD٭1LE䧟~:I[ /Mo/;BUާ5;=ȵ9I'ոgÆ s鶴/.m?l뭷NݓJV_N>߯z(|Ew9|[+}ZJ;`   @IM\iT{D`r|ϛSWjC\TErGZ_Yrj;`=&3 zʔ<(ɐ!C)EE F(kMȕC=.lj hb7f*ՌRDz*xm9ꨣ\ܯ@z8~Oɰ ־GzYn5YiڢQo\6/Er[mV.*H\;Uo^CA*fT}-Q/H-xS=UEWEsK.aEAqi[k{np:gɾf`*hmCa}:7O<["]ʍc{^7 c7 9+Ik%{ܘDylESMb0Y ~耿;ӎ,y֟s[Ѡxo폿K;[߭ot`=#h̘1sܺjPRCިoVqtoX y+bjQukOEJ o":j9:-*u\9rqou7BX. =kFuWTrۢmt8q@uJ5m4'=ﰗyϞ=ؒOcwL@@@ ,@^3pMA2˖YlxdZu֬Y\ >)Miӫw|UVqN2&NzlV[mU%t m۶Kf+ ?᫔:ߨQS} R`Qթw +<";_a3*ڶZYU?~dSV-R| Y?R8)RK|y7JߛU;ҏRT^qc7|gzYk^:O?Խs׳*i,a}rVNq=wF[o   @&z τ2Iy6@@@@״0G  /5B@@@ByY\FN$@@@@@ h.jP9@@@@  u !@xoǥ6oֺܺew&@@@@L#    Rs     Pi@@@@@z#@\jN@@@@@  @j0     PoכK͉"    C @@@@@zs9Q@@@@@Py4     @ @^o.5'       5F@@@@7ͥD@@@@@B2     @] @^uD@@@@(@ ;D@@@@@Hȋa@@@@@ju=      @     .zP@@@@@"  /4A@@@@]k6     E @^$h     P׮Am@@@@@$@H@@@@@v  ]׃     Iy9      @ @^A@@@@("As@@@@@%@v]j     P$E0     Ky@@@@@Hȋa@@@@@ju=      @     .j'NXΆ      ,X @@@@@  o޼yyp6     e/@/1'     '@ڷoK>r6l>@@@@ )PgCiӦkam]!Mr_J0h {饗o6믿>qy,=z؀ji}7֯_?OcJ+mW]^ʙ 6UaSB}ٶ;ط~koyNc9K/mO>?@@@(K: ` 쯿r=+/첲8:)ٳTM7_r._y߿ڵ[b%uSn;ז>̺wsg{Q[Ve= Q!ClM7~W_}>`^^O@@@@ Y+>]kZ7oϘ1>#[L Ŝ9sl%_V^3UJ;v%U?5rz衶2˸ySL#GUW]eSL(D\Kom,袦 ҙml-05g-!   d'@<;:ɓM-T6h#S/ֆ ƞz ˘Y|/8dO<1v~Ydkڴi+Y d5u>wx޽Nʹ9~7F]+|Rl   "Ճg5 F@ĉݠ)'pxkM*N;gO铚矻\A +3[oEJWzŭVWr(@@kw(R_|q[s5snLUoاzUN_ՖZj)w;V'=ػk}͜9fVM6:ud /p毽Ǣ W[m5p +}׀ N{nL}s׽B UF3NeW]?oo>kҤ)`͚5O?d[nejm3j)ꮀT@si_guܴG 2IcgrI6mk: ؞{h{;puԳBiL|Qwgvu׹߉g/vans9ǥJv[C@@@Z$I&gR$UQn LG{UaӧW_}6]p3܇ )!<#֢Ep:9Az曯7Wk~xl v@_|iwy v_1bDe 9uT;SU ^j'_iRS.j oDE9;pqVӥM%ոEiE:,;vlt.+V_}?vin^։>g Qw_!㡇r_u>_ͧP`z    P8,vmSVY=3X=lHUORa`6lS>gQoN8uv]vq-۟*ݬqn!F k4HXr$ka}O>cXc?^x Κ5Rn׷`TqǦ4 ׶/>*@TRW}[MWנInnt?zGo q%$~j0=[+*뮻ZiUUW^iԠ;C ԃv;f̘J5jVq:ﾻ?ꑭGüխ蹤4+?t[tR{|C@@@YrZpUQUD}ZBi4L*>K4n;|]1.B?hR73G _9x~5>7QQ.|T:t2= :ewO?Ҧ)ة**M+Jۢ4$Af'6Vsh73_J7T^{mSXճ5ߣdzr]iD|}R=^>y| 8}oON-D:QOsxNT{mۺw@E/Ii~$}    \ȕg\AqԳt[\:DػTeYײ_R' nz׷j*\䦯J馛ܴRS;Z<3;. o寺+]{wYm\S\Eģ0Oz*(jg<,Jsr!n[^g)) 0)E|@XGnR=Ǐ \oN.@{_Oq}ӹ=V5WQd;1԰/v_    .PP(Hr^{^chRjζGiK:}ٷqԲL-5kr5r[`СC3dK塼JcaRiLTc}f 4iK5y_T4naj& wus7V0YUnFנUY꺇5Ȭ4wӦMN뷼馛.ۧ>TlJu74~td]}5he3     <ȕC=U(Wʌ|> 8+@۽{w`+_tv:esqG`S(+׳TѵnF(zYz4ح)gT||Wde WP]3c^{mwp4|jdy?|u]:R=U\iLUt;wv;}GpkݣQ`^.Tm5Z)=c@@@@ ?0`.⃏q:u +z 6 W=zا~f+CƍݴzO7ot>rS0T=Ur)-ܒ:Ulڇ_O ~Ǖ{{4>r1^quWʺ5Ȭ\snՐ!CkqR`\N;dkݣSs_Ez>oGE秛?    (j\W ~|B)?ttQza+)_}WM FePO ?>kGW^9T|З]v3ΰ=3mQ S7B/rtvʏiȧMye˖mVu+89cWV#j袋* e<*JtwکVHL%׺G5zhS}Rc^E7(Ez$׊>    PG MI ބ \`ϋ~~./+I((O=UU@V;Hw; 5/YN6HϾYa5W8<)g4<|ygM(@_RU0}ٳ?[2SO87/z'6>z]m\{+ٻw]?svGyz[~ȵ0FmdW^yelz+EMVu*e}E[رjto7kݥ}Θ1^}O2! T {IiU5ZJT|*>y9PXz4    FrUYAc9&U{rKXU`Kճ\};r0FxSPmr=U@,,J{*B -VQP2BDAw=ܼNwc-k7oR(R|a {i+PۿM) TקzT(\/)[oe_|q*ōPA^CIj]_]g]Mi_s\T8TCI4xzcer_O>9\dK.5j:^z9JUwFWh?uvm@_{}nߺYjPx㍭YfZ-iiiM"w]3ݳPP\zꫯz衩lٳ_-3eޥJsqX( r+ئ^qEijV//n5ktM^jPÆSN=u}̌V^ocjZ)a@^??5Pb Æ :}xSz{(<

?^==ZJ]w_>ȆrAHre?+8):ur{=kܴ~z.č'7Fef@Rɧ~_sq u6@u>n8օz7@@@@ },8qK[JB)R(%.g RSdsD#_9UqCALEAIֶRN*J63gJ#ɬc+ m3M.䧴5J%sW*lvԆrJ5g&M{NRԅ O5 TUh\z    Px J]J1ETC{\    P3kƕ")1uTDt JX={{3#LY$    @]     =uQc@@@@@ /"@@@@@{5      @^Dv     P׽kF@@@@@ @@@@@@  {׌#    @y     @ @^5F@@@@( @@@@@'@]3j     P@d     uOyݻf@@@@@ .@@@@@@Auﴩ1     @} @^@@@@ S3N@@@@@ E@@@@@r @[     2     @s7c @@@@@2 @^S@@@@@]yfl     P"r        ݌-@@@@@@y\DN@@@@@ w乛     @ /)     .@}z y,Сyl&    uqƙFaEO?ze 6y{6@:(=ztӮ];wW{yk_%PoQ}3@@@@ Ԛ?cW\q]wu>[hkFUvڜ9s*6=+5Pݻw3<[lpv ZqP @@@@$"@ޛw3JѢT.b*mغ[i>3rjвeKM=kS tP@@@@%.W\#g>rH[{kdӺЬ5 &]tK⿗.OK}p|@@@@t%+g3 hΝvVXaSVv*kf[T'OsPd~]hF\&=P[b%En~7s{6M6IWԅ))@~)ͦ>l    P_z%oϴk׮vI'٪}vg$9眓B@s^mTR駟UVY% ͘1}}VO=Th S\A6.#   %PiݺuscHk/0`e;8G}d}:nw[֭c1s@]hΫmݛN<<[Gyd8+mZo]\{iJ.OKqgpL@@@@v,@4g}v5h vyo:wvش&Lz(mlk=r-6f{z-|[*m n_~>:Jn߻jfJ'Ovն2l޼ohl?R/blNwݖYf-yU_~~w+&rTg뭷v .`UB[BU^xz[F6P.5y?)ڳ>kJߧWVl-ڪʔ5ޥrl۶y7V5FZ?:狞kOoτo9 74XqUgUS1 f/ BZl鞉ڵsדߺ_{9ҵyY+[.]{n 3_؈#ܾrP"c]v 70\    PwJ נ ĕ/vu׸EYS,iBfEA%,ڴic{g*fmrӣG6mVtM]u-ӦM!C5\&.uyPW;C WSLqwuWIⴂdvKSU/q'-á2.*t\`K-2z"ڥ'5{}WCT WJ)5XEW(-Kq9ե#(drX1|ԁrPO5p4m4W)U=&:jRʢ5R7l6ITJJWJ Y uupA qXic?[%<ϔ)5N# wyghW@@@G% 7a\Ô?X%)@^MysʵܾU{]9zgJW,?g k?.R#Mt'D+ئfZO=kU]k-te'$ZfD=\ y?etןy(g*oBJ)q rҭ[K,a&MH믿^)ERR l_}7H_4L( ~356h\/>}͚5˭4Z[S-,/i~nm~ x5I֏Ϋi^SrY]w]{ҪV. tIn@ƍs M;/O?tl/瑂oFty&꫻P뭷-Ҧ ۩S'w\*qCuon̗'4Iy T7z6XJUg('mɦ()xF~;q5tPX9fu?qRS@@@@ =@ ĕ?4yUviv=Tڽ7xcbmzHǥq|@xԨQ)^ Bs˥ Io7cƌĠqᩘ.Oj@@@@< O l$3K6 NZpJ6Wp;'>1bD8M?s.MLtAMF}wsi@;Eܽ;:;t˪B+T\Ort8%I Pq)4ϿYfTb?{|-LՋђunw@@@E+87g\i]v!Cd<|R/\,.qT/ fYR`L:F>M ڱcGԧz^s5&m۶VQچ^V.\ݲ=tM ?)Ȫ7|l뙴^.OIvmmڴҘ(v:@^=7]7N~')@瑞Jo8^'OM}kgqFst`J;oF[ oAVUⶍ5hg4;IyUJ\ڧRDw@@@(^:,Irjnٸq22֭[ }n_|ѥHP0,R /`ʝ^Pwj-sz!+Guk.vF^M*@ؼ>s w%iW,xw^-Ty4R=+:גK{vVbF!n|Goq6s1ܪ_^SѣG8SAޡCipn_~q) :y\8 // Ϟ=.S:|jǝcu.d\=4 6;Z#@@@% +k\1S4 8}\/\;iݺ ?;8._z;Ҥ !"" (RD ( HQPQTzQAz|9ox!ٙٙ=%y'3w߽ݺ̙3=R@HU5 D)h=f<6PCm۶~sm"3w BZc+ct!h3fO1|u}_ӧiӺ@` gϖ_~Yij?7@۳fjI2@ 6<_6l4mԩyM x38;%J䒐b@x#UT!C5EhD|nc͛Wpξ]> 9HHHHHHH ns5kӥhѢnRx-Q?~ܥ)S|fy 9S1nwmw)[ GGPݺuv iQj.=)ڵwٳglٲN獅,0@TJ#&Mj_崌{\rGr!.;XHHHHHHHH_"3Xy ܒp?͟?_Ǐo_:& [5k8'VZ%>)TK( n{\&       M ߻w|BxE֪Uj۷o 'Lj{zYw˕+''O*2yE`t$KLϓ(y;64:tH'y2OI"'M$+WY~5xV{]D^zO` OVq%1!&#uCœ@xw;xnPr|yFox]zR-& wQ!읠?"|z3xG!n=#˗\O$@$@$@$@$@$@$D@jC͛{ ra9y=.o 9.lyi׮f=ډ'TXĄD+xK&%_ԩS~< b{'f4UN˞ԏ?حǨnBM*_zUΝ+|ۑ|q?\"s?3Qxq}*4W\jDl+,3H[oYJd9n˗/ٳۋe|'mm(<_!V,X@r0qHHHHHHH NV^O>GWu림0{C?#'NT1*;Ӱ~xTw%i& _x߿_7ŜIh7zz3fܹ̐s"szAHwJu%OԩSm)_m@/\"{?yJAGs=;4       G r GR= 4Tӓ-SoyxQC@[Nғ@/}vמ??k;glҤI~͇ J^~enc(D~B8 d#4 wdLoܸ:VݍAh(wf La!ݑ#G<ʃaD {(뾃!cҡC+[w;CincqGRœ@HnN@' \!$<^#k3eI:udw 57_9@p>ӝx>v=ߣG{qߑBH C/ytwOiF9amca^#|/N0!º脃'~dFs$      R $ݻ'ǎSb0D!C!Cɔ)dΜYJ(Pj}`Nϟ?!d?sr۟cóIdϞ] LiHr8',# &>|Xq@!FO^%\\PX(|uPr ~:w0Cb΄ P|ԽkX~yHd$Iphbۀ]K.I,Y`?y@       k[lQ3K$TXZy6֭k׮9?mڴRdI2®]d,cƌ-[6)Sz2 BׯK޼yHnK@X 3&M* 3޽~Â[wΝ{-?}ď'I'J*2zh6k,ի6[nu)cApaM6.;4hh¥$jr1c^xFs!ODq^|EYr)RDI ڵk Vti4ie~ܹ.Ns疉'C=R$@#p{8A~s9ŗFVL_~Y,Y"nwLxbBF ("V?cC  x=kȑ#eǎrG}$769{:qEU68-g/:~qvO-:='!+e#c =3ݲ<3pIrҠbۄKEInTs&O,%JiNUV\9)\NZtⴟ۶ʺ׺9rIEv~=B-lT \{F՘$y(Zk}O5TY/\gy̙ڱcGYf=H.߸HG'Nu謰Ny_9r?9dKSr+sUIREg"<[Weeϩrqb0Ϝ:4^&W%I,mۇJ(ꫯ):uJ>] دpK5HTi^yj'"I3e21siYUVwD3w@NXi$¾l:|_v1K(+ 'Z/*ܾ{[ߵۏoVY%irHU%G,~;Yv[~aTYݞFa 0_={ٳܹsJ"{Vdj?ĶruM*[>X5%e~5Ųo_J'Jv]0 ue˖I}o1|v j62[U;y^i @ߢ ['cW}D[AzJOŰia^I tMz 9߮&G '˾ UxI0{f…ꫯS w|mr/3/ͬ!ɳf31sinC w|Or=i#U\'z>_ƫrjjv@N*c@fyTEn\&E wY:ˣ>*Ǐ[W.g򗆮:׍~B% _H^8wd M)%mQ)0!Hop &<rOd/G xh!d?b@ş| I2# x`'W*rcdً@wrg90uYf Hh/tHy~\lY%v#^oD1!D.\xTWbE@Neea*< ޽[RJ%rR=z"1 iCȆE8Pôy~b,֙q k}` AT3k]=ߴZ.C8v<.?:L[>EmzkVC2L찥JI'l) exë_\vbunx&X#c =iZ9r52,s3_r}.n^u'Zdɘi GΖMﾭODЖ!]TG ׯ21|lr2p>ŰZ6)tõ3qQ rd }%צ|H+2 jGVB,##8G^?մ{Gq)سPvnX9KQU tu$.}]fٲeC78c: 1B^yQkow+Ud/~[!&{ǎ ǚf5|}uZjgp %kgh[YoYiC0.Tѓ1ޓ!yI\Xw\˼G̓-G($p:\ ӎ/g֨Q#OH .@mbk×O|T:zܭN eKvES(\LҸqcTEu'syS6C7+=||ch6cW "iΜ9*6dfHm۶M6/pmуfz/N tpt]o)Λ0wh۸RN3 px*W279+W ^%N?X`j .61$arIm=9/^eh_ %Tsі` zHj/[*f??ICJX~ϕg t]ijfx9sgI)?XeqC7xzx]Dl5<uO.+6;}n;%?^aXt὞Έ]%Yn @Fz~ޢFԅ29_ܸW/v(zWA˭|G(2uT>a!VwU2U*)r呤h[hc/i *ߎw: N8bg  °^S|Ṵy-x wxQ 74W-g^!ɳ>xK{wv:Б|Eqȓ¯-q̗(3A?xf~(n\~1丘sV#I$)%oR0SiT$IZoO9:!Y||Yoܺ{QNh<Y>jޥ0Z>2Ndp&vH7o^~1%w.@m4h xndxû~OcIJ)bƍ,?|.pOrt0g@Q$GKhXNY%7ZizUo` ]u$I]uS5OUF-oԉL2{Jyj%ɲd5cfAy%ϖ]җ*-)rRb}{Mx+闒B%̚fuau=(-]^ 摈eV_nbcd^:,k3N;, @,, @ $Rف׮]0}f|^S?6͕ թSGAŃBT +?G?Ѵʃ 5H6{lի>sVź{$ ,21C`xXBdɒ-`!62խ]Ԋ is=oU ԡ$^ڿ8DPxÖm:./Z#|)[ 뎨7-!mSVVe'M8⧣bt?By0.ZΩ{IաZ#<_eqg  w_֤6lP;O3@$KH/c\&NEXÓ$QTz9 ;KΆME{[oʉk맍>`XҌP,Jh+b/z+µKlɜYFo' _n X]`}Eގ"4:޸N?qt?qb;$1tCfs=\cֻcwX_;εr%K]ҤFxL:1|??ȨL 5 H{k)—os8DrDdt8P>/%CJ|CYZh? "c$ka|IGyU Ȇ q(5ާaN0)uUaZTΟ?/OWt@?+'18LSՅS޳زeK-*F%_}Zsb!l￯7""^y{{o)]g(lc94APwD3A%-ﳥ/8:Z& w{ݥ[ $g gF(j0w*#Ya] /9`dyjfxw֪#\nKzByUbW㫻v2tj6py==(vFumXN$L ߻w4mDq94zq @ ~~ BxG"#8x8pk)m')3?ъL|&y9#Ԋ7{i*#v Um[5΢!g"G8U EԤ*=j鑼-YcR =: {9:p#?S~TK IbӺ|^eɐ<-kj kzU9׼^TzxS|yu5k/ln*nݒGMO&x;mzyYWKVm݅O))+ŤMGVr57Nj8^uYN@@^wBI7sa'G$^BӗOs$$^$QËGc.9Ջ>4uYwjG L Z^qI6\iv@>|CLGЀgIU ˤs"*xg3bxN99B@U(\xhcЙ/6 ÒaZvx$#D Fnók0% jw g x렏 z@#Va 7^~fUs<3<\Qç8DѱI=%̓WOrӗȽw}r5XTƲU O[Tf,lYr~UV{, sH_%{'}#]::UAga`V4cGK 0%2?ҁ!n7ߨ(]=[}9#˃7ԪUK@?F"+xxЂ=<CUdv mu8[!0?9޼V^=fl}9s)6 >MgnuJJU vͲ j;vVqK- 8%D{,}kbY6ZsN@R&qt,|֖_%_BR(K1'([-i u~6+cF&‘fNxyyr+C,t`n&3 7Zb7EŪ@roBaJPQz6**{"K,Nٶm)0@~i$3gw;jrfdȷl٢#, a|zļ:F)"' 9<ĵQK<1)fU~ئzf2 Bȑ'{cG̕Pi<#GR%#dGfѱVXz e>jhޒɧOo]$7NVOվ}'ɕ|yKUMp@i+ӽ?F#1¸uG _%AX_K|csyFֿЏ=l9k(|~zHns+ @ d9~P Ïxʕ]̘1C Šp] k :RY D )|:Ԁvemg7_Ko:b~ xۻK32B@wg'Q!k ȱo=I1Tdw 88vXvwKzc 5MProyѨVƂ?qU _`!:"$e),P ~Y¯fyze# XUUddHJ[rt-1ǴC !c@k;nv}u`ϥõ⒏Xij'ĥc1V$kIA2p;gˇs^$6گ"߿_%TP,6 ړۺU ;?o]#/ zwF>=ŝ@~P40k]ps)OLqܣXɭdz>sE"QobrFMyDwTuxLox3}[1,n$ՑcQ3G+Hz]Ϩۀox{{38ZRé&A 3c 6@):³ L [_r6$Wf?*M6j)ϯ=nT&!_*Uʅ5 KNYH9OU݅`ŪGKHR3x8C7on>h!ڇBzY…xj73z˧/w[uo?vut ]&uWnIz,UJ%K;gs(}&@p)HNC9)3Sxi CϹu.z/f$&$P 7Q Dtn=K;H? dk0ve^Ma}*3CbŊa}mU <vE1Urz@z0*n#z;MIa^?ܖ_V\h5TUJ/˳_tipFdJDž.u¥۔ֲ!dp/-_@*= UV$jx{wx„};w]fۡCxZ=6ɍ" #|{` HV5ε}?J2u>QY@w!җ,--\yxdM0@^qh#|Ji{^bJO p]*_6Ghb/:0:5 @#2mA"G6A/1٣, VvGV ǐΝ;իաcZ} ߏEw>?b;):wP@1V`dxHζNye3Ylqg"ߐm_|'T 9<1z2ct;]Riwm#=b%Ok=]T oOdwߨhW-vVI;!)GRNI΢ =!!"ȕ+Wԃ ©Xpۚ#\r$>͸1ws$I'*DTy({:l,vaİQFiӦIxE@Dɓ' D XK^S4|[4Ҷ~~}*j0G<\"%f$bHPS&{0DXwGz3 {IGqO!U0@oOĦQEQ![=!#yjڔJظctrd^4~(̎ܬITzuUYOt#_uta 7\ڪ9^K`MnxDŽʢK 36mQUv҈reu'KBk7ȿG7w^k_ol![ vf].S@6هouf,S~9rFI3Ա~nv@f3"c9pf<"JƩz1Y6uf]n;ǧe)Go&r|zw|6:Kۊ]usv݀FFrk1m>O%_ncO FO',?*~ZsuZ'PrFm+Q ;QT+?XR]ŧiJ5"+ qvtI98˻|#}[mdCv~P{a?fyt]~-{Əы :3QfɂԲO#n|s}ٲݷU1i#'^RƺF%͘Q$w$}ND @#2h7 !\9s"P ץ֫ҿmIiX1$J_V{Lpu֔byOy&}^2+SJ{^Un2ҬZn5Cb /1<Ó&vDu^<ͱ B`}L]|@>< .ݔ[F}m[_QW0dFhh7 "9αTT'W{&[mdzÃrL:ybãC$Y*mQَ{ߗ:lNz jmTy(^=,i:t)TUlMT/ &w?἖&Q7p@/Ҭ3qH$B;{څ^"ɲfG'3D8X֪#y 'E IgI'!}.XA4H~H\U+ֳj?z4$]ش>G'${ψ\LʣܘjCO$e(WA u~YRkc"v\ܹC^V(i >7 ߱pj}0+ro$3 T G+A[(&fq]('wD#)뾫ߚ@uᡄ{eX"<t"+WTuSOy e h5N|3u~=Bn}ZCm朗=ً7F7:O+o+婺*}A݃!*{ȭ´mm* <8KWoF,[NZ#jDG\CA랕ݮӅ|w,2aV z_y.uUQH2㮳խ/G;:(;c+ahBK|.v-:_T[T?>)wCdl=ZQm`a"P]Eqn4mxXlV :BNj#o1z]vՂ[Z"^>}z۷MV:uꤊG!w 0tTxNC")J,i&?U}v4hzvc n׽s6n*8F]\B6~]58MZ5ɕI`nx記S 9IuQ \f8̕C~ v8qH1۶ʿ"2ۃfؖJԩ᝽eȇjFHxk[ Okz[y?d6U.l$_d$Qｓ}k RTeWu #aNr#$AˬzxDr⨰F"% Yֆtu}`L/b1rH9TXtݗyvCW 6-jҋ1bpO]SZ vaͺ>gN]:.'.Ubn\4Te\_?8j9w"R %CJ:r¡u^84ϟCb#ѭhݻwoS=GyD2eyaI*jr&2_f;f#aSp;z db$I$MB=a@u/ +m7=iF̂QaQiH>Gqo $myx`H vp'G~Y_ڼoҺn*n! Ed /t:rZ$ãE i͊+sLH։1_ӧ 駟6=#3~l.;ޝHU= 1r(Y] :TR[_u7o\tRɐSs#9W-ZλI`8 .C%* I%KDzҥKN挹Ĝ)~ ^`xv"zTXLg#*fR#7ohx-/[Ly-?C+W.Ax)ݣ&4b$p?2glj nWv]:ʵϊ:擤MNzZ8zܾ}[ƈ BǝG"p%@<\ ' =fL[v0>$@.sn Bh{eQШ^H b"+=mC-x饗}茶aC &3l{F!k=)­ <RߏsIYqhI_rf%3m^oRkYL$@$](Gyl>霛.Et$_v[*4iߝW[q* @ڵK%!Cqx$ccrHQ ͣ3![l1aۃ}71ukeAb #pj*YytAO>L#r%wZ YR(kEV  ) !̓@h9rQsueI|;w5ɽ"I"%)bwB" @ p)9n>$ϞS 'f8Xqqy$@$' ~cu          Ay츎<           ? P  cuY IXHHHHHHHHHH v@;#ςHHHHHHHHHHO$@$@$@$@$@$@$@$@$@$@qy$@$@$@$@$@$@$@$@$@$@~@'0V'         (ǎȳ          r?: @ @Dۼ͚5fnJ-\    MQ׬YSiӦ駟Fyu; y$e}gFk0yB-r絝>xyYMqe';G:7<jN֨%ʼWd˖V.)f0%nc(9aB =m5S{@mdn Eŗ+Ek sHo>R̙ӹ،:W@;~GeҥRX18qKx@@@A<ޥ0c(˥u挺w.= ;Er2)<r=;O9x򈰛#?z5EEɔ;dp^̩L=hT=RyRcm7HӡU+eoʂvw 9\wZr.+"Wp+ZbE'S4ǿr rU清3pWܽ˪#>G[_?at(+gwmr~VpMuTns}8ۡڶ6њ3ѣs0O @r,et= [/J 75jl6g9i\<{| Z_%] mcGe)m}Elv~ya=ɾEM a͢A>yW|q{/s=kueD'^9}ٷؿhLؾ(?*Sz/HkIa&umYjje\1ekmu-uz :U=Y?9Lzs-xX~Ｙr`_̋n*mT>2o\'ʤN!gSV.)Wi~)'v4J>FN[76YXn֗|QYwڵ$C%nn}ϝ3 No~7Eޛk{-zm)w{ ﳹ6ckHE\w@;nժZJe& Z|: :   L "Ν;WrϏї]\Xcǎ2g_]}.Yɓ'͛7E/_^mw%͛77u&L sfl.Op7߬/Jy>xG@:[ Ѕ:>'%[?sdUn#k+YRr ė+g:un=ĺYNƶ\|=i4ɘjJGjt[_|.+ۡzqµ0%i^{^I:ϟUxJJ?۲XإhPi}xUzSVo\x sA40D:t@oN 0Y ?k;+\aOJ[y)i.}z΀\czܹ|Ys_im6D$rAv@_ۣ6w"kVXSYo<ֲsyBtu]xf- *ҍMr!G}$C 1h   @4 x@~!MPK"Edǎxbٰamo,.?cNl:_v.2{o… 2!k;om۶hJ_On}vlP\JΜ=/Ɖ}T= 2Ri7YپplҰJA9zXSNs79I&+ć؞xj@~;rA|o1 _yعCL/uhkz%5k31ᗏKp- 6 ?~k_]w!T"Ν=sgFqdݠnz:v.֯5#39wkԴ\G\jӑ\c?lSu(:߾}|$uG u4;Z3Q/Ɓ6kWY5s{w)trUYe3 GG) M33q|M u.=XsqECLΫ~4π\CǎBg!_AD=Fc& ğ΀<[f$Ntl[.u qǔgӽ H/x#F04=t5bSL7'g+C@A}KvLd>[rޠoig[l΀^7At1o=b^fȥ_c!s8yMpK=wݻ~v.[:zk 4 w]Cn߇w9>oeާOޮ֦﹮`w3يp[/Y<F$G@6HN  +;##F0y]wִj]x ~'o}ms~f͚&dpO?ϹbܸqbZ>0+y\7 ߸cט=`y2n{=w\Kn*GU{噏7˪#_h80TKp]$g@pf4jЧ-]=#GSW 2Ok `^#̵y z 4DE_ѐnVktw1Xh~{cPuWC[;m^n3 yP{ &g-MCc>q۶uZaps34uKWzGJbM6 5 ~ΚL!zJ?WZEx/Ҭ)5fM=spze/^B o:^AW;=/%zĹ:lϚ^mf-GX#sHkh:}π3:~ ylD*'Lq)}ڣ *W)gn'90R_f?_ Νt r}=׿?wH *nFC}m鮸ZrFRiYǢE]vf / ۷ӓ   @Hр\k(MG~ORb猩G7 LwϪYSm wB`}H/ǗҠTݳi=f @ WlYoE?R2 {^ɬg Cs])Pk'!> W@^Eڅ3gdJ:^wS7_lJng@^RM|شG7;.2#G~"PnxvEBg{ˠa5~R+Z'|ZQB[S`.yI ;? Fa(ag@5k=kwn%/Z%/<pnLqH` rNNwŒ^6 DZ?_kxunèdҢE m_[5ث 7d\̹u]$y~=os*eeK4:['FJ期D{ѣGJc   aHр\GE(<-Qj$_HRr=us;v߾}nݺnbҢۯnwcZX2)r -R:7~/LJkΎV*ڎ53 ݻϮҠotF<װjv]5 sjonl$QWg0wߴJaL7 *5s2A|Ў<:E" OaVYκ\?-WB# S ĚR['>G;o8rg95?3MF};'NAkiylg,Ĥ3[e~{IkWutIBΉ45\Kqcf֛OUW8r,YXy.;yC2o ٓA@@BHр\덏3Ɯ)SD'Lj % cvU~%|_懲 dvg#VɆǼkMWZ /nׁrۋ#G︹{D=<9r u649o(8ƫօ38stOˣa.ӑMZk 㢣e-t)oZvZ5vIm ȵ4}<j*uJ `t>ZZnQK_^`r-ۢ:W3Qe0fljI?f[7^4yL.CXDwmϼ9Kf]\;7Bz I`?LPgjر 3,C@@R@:;OاOiРAM8p4u3e$\[ QkRw'i VmeʷʬpV*[yymUs9>G'Ds}K~ ȋl%?7 /Y)JgPkyqsmdCHhכkMH,Yk֬lذAr-sEH;dc@@@ R4 ?eM$D6|p=j7{1Yt3fHK˵n\C&1%r~kdzZK_k+-R@[mZ?nj?,9 Z}u?X7ͮK0o{8kW_@~`@VRy$s|pxN%=̼LF\r#)G[@6k.`&V]Zؖ;kMs̅@hymҤٺuk֭[[@@H])Ϝ9S{9o! 6|Vp~Yj oWx=f I>E'e\pkRJm3gl#;u$s1(qI _"wyqI{c~)'\vj!7weV;@[;ZRb~9} Azl? 5,YٮyuR+{FLќ=˕%Ŵi=򧞊)/\eJ)иd&Lo7m4#<5m{gs0װIGdW^)'#ZD'M3h\Ussz<(զf)P-vWG^}C%dh7Ɏ)M9뻼d&uPC?j \sXYwz^,e>@n4A64&>|M^ΐX<=Ve%\[\s'yr˹O'6׹'dAÚͮZܕnsqe>Xi* H?kzfM_T]dׯe[ixX`!ͬ,[봓.Coh~foÞf#G'њU*wޔc֙+Lhף֕5Dz.[>(Wti]&]칟sl`&R%{}&fW~ w8.$ֿOo $~QDK@@@hH\#}uܹsvmVem߾}rj7N}T!o޼ѣlٲ.\p-2g#)׻K1kk6^H@}_ھ^#7W?m۶ŋH uxbZƏs=k5qˮʹy h+D}xN@wryq|+ >-y93NBggjkgKo{9::{>fV$&{2CZ:i3Oi_`\SB{nYe0.ݘ\|}pRE5>\Suƹv'])Ckk*a ȝ79;gݿ*uؼvR!ȅZ/@6M0}tB-[HŊ]ٞ   @jH@IuDgӺ:tns{X0.bwp::|֬Yf[o%<9ʼp¦ƶHwkךI>˞}YW~*?IVl8h/r{NQyrR4w8nwi&9xMGltUOʦ?xnnFx)h[YN٩"4\ѺHɇYؖוjzL{Dj#m:lBF1ǀoKmp[=r,\9#6 MD}s_h;sT:vW]nU5_X^^~.S!j#7PV}_VCo8[;/勉>כckZZvC}%Ojn7mIx蹯n~| Z_ od(-i4f\qMy^O$o7=prj߸-cFZOwR/9D@}W@}5 |Ҹq@@HU m8+:HXɓG4R)i;qVGNYPެrU_OڵmGeRpNڜVYtbOL@q[Ss~GS#mBYOXjiǏKyL9VJU'nfk'޵܂`)4Z-5Mq0ECSVm[L%]^[ؔ #'I Ƿlndd3u3^_&#rg2GրfPo at8ģ7Hr)~}YL@C<#7.,NF@k/_\Vj&v4@@@ "G#Zj<ȝ#uӺu/L>sT/޲U#!@ 8G1B*T   @r 'l 7r<93?[6aR!@tK/L:WkIߓ^Nٕ͔+X7I$;`}*5i^Ҿ@@y߁0?yN @ hI۷ϯt- `u"Og{@uBۦ+}@@@@~ikyt_-iI5tZnTM;)tGe/tsGy.u,E' !   `>~F,MJEgƩ @Zx>Ś<5nnU\wYn!   @t G"    Iy.U >s1-sI&Md֭ҰaC8p`@.'O48 w|mG'@@@@H" ye&}ydy{fߓ'Oe&q;% wj$s _~;/Do/1@@@@@ D$ СŻ}vٶm?^,YӧOJ,^'zH"ym mV̙#jlٲ_^n6uȵ9 @@@@bW "[o%?K2x`ٳYVhQ_ ?]; ISRc<ɳLG?ҭ[7.n֬ZʔYpO>m    iS U6#<"7/{%-[W .-gϖGv)Z[L2uof6ݻWF_%f2:Yծ][T\dkkQG.رCuͱ+T zKk;],Ԁ\_+VR_dƍ%K?C^I9ݻw֎צG3ҿjժv}fDq͛'k֬uIRDћ"5g{=l㏥o߾f     :RU@/kƮ]ݾH_z%ygFk03t2Q݇7N,GBo75s-~ɢEdРA^+VٳuI=ŋKV|D}C4tPS<={Vz-ZZWr)\_BW^ѣG. ,|%iӦ_f @@@@bS UMX-l~;ԟnР}UW?řZ\TڵKtı40kyګcƍVZnF)2 us2qDٴiYWR%@ڳ9I&%Xb%C ΀iӦ#yEץ΀<~#_o tك>(~<:]pi^P!ٰaM�﹎znIi[nlףT!    @l \K\wuwBTgɒ!CHΜ9H̙3 uF-{MLg͚<\Rx:5;vhʝr_A3$vn빖K՜׀^Ge;9rD;3J:SLf7I=wg@޽{wSVEq-AMjɭ`Li~76)9j]-Z%oޱֽ / Ǐ/+V;wg@5ܫUfv| 7Q:ݮʻkF5o܌fZ;H"*;9IIDAT]%T4l:*^MG{fh~    Ħ@ 5˕̘1ÔQ:ܹsV/s lׯkS 5 3FN>m#u6/9\i_-cƌ^#}h%̹;r 5P֦#u$DKЬ_^ҧO/gΜqmcvŋSS..uҳgO׈sWɞ={\eo,>;ܾ @@@@@ K.1n7-2sL3jY%ԾiԨ.I W^-:9O=Ӣ:3 Kv3 LІV3 wެБ֭6mH=)Ϣ.o#4ib'7xëPYMN굡r}i     * ב:٤6YCb`T`k]rmGug [kjI ȧL!nS1_|楎 $ yr}%# ȵvw?vy    )rg K,1d*!~W HNp'Np5ȵy=S" O\ ֓P=Ԁr5ykJ.VpZvZ,3gk ]s_5Ix[lq;0˝9aWǏV}xײ_|ѵ}RXW\sO0JaM̦E@@@@4&jFk}qn)2(jrݎБ:B\Β%kDGϟ?߼LhQW_5y*U]|tdB|SWJ><5 CAvMyS˴$ImV/]t1ǔz v_2=β1     " $:1㯿jD5`ٜf߾}E't6<{ln j:fL_5RԳ'5>組rȵ)iiWn<:q5j0|Au\u&5yٙ     "$: {۶ma?U6#cƌ_~,/Ԯ][Μ9#w=|׮]QZ747smsv/n0>o5kVYt9ݻwj| C9p:q۶m[yǥp¢g`ʦMdƍ6GA#ɵ& ^fuBaY    iK "?B9HѢE={քVZg/.Òk H boW}YeXݖ9_4kLfQj z:b_=\k .~llN +W.r@@@@ ȭ Hk֥._.]Z*U$䏒.]:gZEK!2OmRpzcB˪nzD˭mv>h 'g}fnnq4ۄ     2 O VQ̓q%ÇJ*Oݥ]v g%    Ćyli*ϐ!􋿋Q۷o7unCC@@@@     ĤyL\4     9@@@@@ ɷF@@@@ 3     1s     |@@@@@bR<&v.@@@@@     @L E#    @@@@@I|۹h@@@@@r>     1)@@o;     @@g@@@@@ &";wNVXa˔)#ٳg7a֭~szV^-OB IbҹV@@@@HE ȷo.3Æ ڵk"=]ʈ#jժ2r=X*ۻϻw.]Rq:     #z #     pIK,˩SdСrE([ԯ__ϟ/+WL֭[KeԨQrA -ȑ#<=ܹse֭-[6i޼ji֬T^=>m4sKlAÆ to̙3>,>5jZ߹sgɗ/r     HUVx]c '5k j[ܲeKի2x࠷5k-ZT7n,6m jeرc.|hBKσ~ԩ&`zC6@@@@@H{^; 5Ε+۩̙S ":\GŹOE6mG&$ׯ_bݽϛ7O *dFxXk}B ttWnݺyuݰaz*U䵾CҴiӈn߾׹%@[F)hq6)UsyާO)\r     HIo߾ߺnذaRvm_2jSz@@@@. G@@N@E@@@@K(Ga@@@@@y> qX@@@@$рĉ2n8s*7h"Y~fXj&LcǎIJbŊt\+    "ȁSA@@@@1{ù\@@@@@xr>      1)@@o;     @@g@@@@@ &cm@@@@@     ĤyL\4     9@@@@@ ɷF@@@@ 3     1s     |@@@@@bR<&v.@@@@@ ̙3;wnɕ+W̼#qqqgI>+VLҥKl׮Ν3Ȓ%K'߿_=*Yf @@@@@ D< ^8p@}YyWzqygFr]e$gΜv:uI&Ildʕv@wK/رcbŊ2~@7     Vr3r?-D@@@@ OI@N@DŽ.     ɈЮ |@@@@_<}     b1>|X͛' .;v)_+WNNɚ5kG* _bL8,YD/.+W-ZHRyΝ^p;}I͚52U?k,~A@nQ    Iɀ… r!ٽ{?~֩S'9pTTIƍZO"O,נp^}µ<\@@@@B\'{%..ͫDכ6m2ϵԩS]˓Iruz%ѣGNd\_l̙S&KUbE?~|"     D@Nq瞓eҪU+<-_}~nׯ7֭k&-ʰm @@@@&zWq֭ECrmajՒCJ]W3k,y'-[#OF@@@@@\14 נ\[.]{bŊ=zu#-z}>E;wnsݝ:urM^:yd)[vKF@l@@@@\'С_J*Çe֭i: ϓ'F'ƓO>)ݺu\a'e      fܸqҽ{whi۫aÆgK,rɤI%V:w,'Nlٲɘ1c䩧27.ڵs.JFˎ@@@@@  ԩSyfٶm,XPʔ)#3f ,<]S2 u;v5kH…͈ 2,ȓ"    @1l#'ۅc    .@@ľ@@>V*V(Ǐ R@@@@@ ȓ }6X    )#@@2^G! ' P@@@@RT-  ҥs     "o߾]իg8 &kNE4{*]v#FHժUeȑ{Tw}w-]t1t@@@@@ F#FGa@@@@@y> qX@@@@$@@9y>z@@@@. G@@N@E@@@@K)8P6o!4iYu)Rm},Y_SNСCŋn{QlY_̟?_V\Xw[ٳ˨Q^Z#Gy5{s֭[%[lҼysҬY3^z}ڴi^؂ Jҥ~3gzu>|Y}jԨᵾsΒ/_>,@@@@@)byVdAl۷Oj֬Զڹe˖ҫW/yeAo?k,)Z4nX6my1׻`Ѡ=ֳgO]$}ʔ)ұc`O]>iѢ|ԩSMl    !byݽFq;vRpa,@@@@@)b޾} uݰaävھe y`sQ    Dy,r}8,    \ G   z9atJAuSqfb$hdæ&[ ýCQ;&BsSnZ2n,)Eu:g9un|wgu~ϳ @@.\v&@ @ @=@ @ @rM @ @ @ @R@@e7i @ @(y@^WW7oٳgnV>V^]v~E.]r3k/{TUUf&J @ @@y < $9t=?׋qwO81y8#cӦMG@2 @ @9M@_dz>cǎM3YGy$wP@ 7K,I<'M @ @T 7yK띷|qꩧʕ+ .[n%. oisG @ @@+<@{ｑ{wމ^z ;_q$@ @ @`[˗/O>9wyg5*jO @ @@g ΋ 矏v)W @ @ؖ<s=_}f͚~x~7RB @ @@< _fM :4S~{-. o @ @ ;y@~5ٳ|7pA @ @@< ON\2ҽ{n׭[xK.K/~?>WTWWGmmm @ @$ Q@ަoqŵ^' o7 @ @(Iy[S&L]v%Am-蘒  @ @(m oMH9 @ @U WҥKO>i\˷z+͛^Ԥ' O:ӧOe9sf$'ϢEG- ۢ @ @Q Wc=v[q1dȐ6H%Ⱦ  @ @U@SOŤIڤƱۦT.#by;2v=R @ @" * /a-C;q @ @ " /c{Jw' @ @(hK@. ovQ @ @"ȋږ&u @ @(xmY@. QH @ @@E'n. @ @Y < _fM_ѽ{j%gӦM~ҥKTUU|Y #!@ @ @ b @ @G@@^>ka$ @ @ Y8O^uo/x[E. w(K @ @Y w1c3ό>8}oӦMz_C=_J?䓱^{8}FEEEey!5E @ @@Grϙ3'9H^ҧ!M&N]tQK vT|})<:PwwD3 @ @J 7yk`z\zi>;١rW0 @ @;, Bke]^=:&Møk@@. PF @ @@['O<2eJ 6 O;Xzu:!u[%/WCU @ @W@@؆ ㎋ߏZ~٭[馛T @ @t@@@/ZC`eI?ȓ(0`@QWW}Y2ԩS#9a^O,  @ @؞@~:== {챑\N3+HYG:f͚[㥗^J甜$b=(Ո|[2 @ @ @39]][[O&YۛПÇKղx|{+ @ @r/X =999 /DY}% O;{kҩ_}ՑW̏&@ @ @ ȗ,Yn@1cF 42?) _hQ92駟<@Q E8 @ @m]@lٲ8묳>%zNh#W᪕S@> ɓ 7Z @ @ * oӓ֭KNvZIq}&MѣGD@^T^ @ @ FWQFŪURtrZT~)6m_S]bEtIe?|y䑍" /6  @ @h@nҥKS޽{UW]M={ưaöY^y%\'xb:~E2kG}wuWt9O2X_Œ. @ @M@~A٥1w6HŬx}իWkUw\@Ä @ @ @;찘3gN %U@馛b…u]cƌwyu Y / @ @@n빬o7D}]6TOy&:!@ @ @y+@*: /<:Ү#j!@ @ @B=6F @ @  6o  @ @((7* K @ @ @bnm|;C @ @< _fM_ѽ{`ŦMbѥK*Ae4 6ƍc]wʌz  @ @ < o:W @ @ @ y6z!@ @ @2ق @ @d# Y/ @ @ Pf2[!@ @ @l8 @ @L@@^f b8 @ @ <g @ @ @@ lA  @ @g @ @(3y- @ @ @@6% Xxq:DEEE6.^VZk׮M=O/"w}*OS7W @ @Hy]]])cȐ!eDSܡ\sŠAbƌZO~:̆g8 @ @D@@^ KtK @ @y\@^[ @ @Km= @ @ E@@^ h @ @->h|7M3gNz/߿In7pCO?ӤC=4N8?oz??***b̙O?5+ލݻy睗9~}XjUt-F1| %3 cluster_1 LogicalPlan cluster_6 Detailed LogicalPlan 2 Projection: my_table.a, SUM(my_table.b) 3 Aggregate: groupBy=[[my_table.a]], aggr=[[SUM(my_table.b)]] 2->3 4 Filter: my_table.a < Int64(3) 3->4 5 TableScan: my_table 4->5 7 Projection: my_table.a, SUM(my_table.b) Schema: [a:Int64;N, SUM(my_table.b):Int64;N] 8 Aggregate: groupBy=[[my_table.a]], aggr=[[SUM(my_table.b)]] Schema: [a:Int64;N, SUM(my_table.b):Int64;N] 7->8 9 Filter: my_table.a < Int64(3) Schema: [a:Int64;N, b:Int64;N] 8->9 10 TableScan: my_table Schema: [a:Int64;N, b:Int64;N] 9->10 apache-datafusion-python-54.0.0/docs/mdbook/src/index.md000066400000000000000000000041261521727353300231310ustar00rootroot00000000000000 # DataFusion Book DataFusion is a blazing fast query engine that lets you run data analyses quickly and reliably. DataFusion is written in Rust, but also exposes Python and SQL bindings, so you can easily query data in your language of choice. You don't need to know any Rust to be a happy and productive user of DataFusion. DataFusion lets you run queries faster than pandas. Let's compare query runtimes for a 5GB CSV file with 100 million rows of data. Take a look at a few rows of the data: ``` +-------+-------+--------------+-----+-----+-------+----+----+-----------+ | id1 | id2 | id3 | id4 | id5 | id6 | v1 | v2 | v3 | +-------+-------+--------------+-----+-----+-------+----+----+-----------+ | id016 | id016 | id0000042202 | 15 | 24 | 5971 | 5 | 11 | 37.211254 | | id039 | id045 | id0000029558 | 40 | 49 | 39457 | 5 | 4 | 48.951141 | | id047 | id023 | id0000071286 | 68 | 20 | 74463 | 2 | 14 | 60.469241 | +-------+-------+--------------+-----+-----+-------+----+----+-----------+ ``` Suppose you'd like to run the following query: `SELECT id1, sum(v1) AS v1 from the_table GROUP BY id1`. If you use pandas, then this query will take 43.6 seconds to execute. It only takes DataFusion 9.8 seconds to execute the same query. DataFusion is easy to use, powerful, and fast. Let's learn more! apache-datafusion-python-54.0.0/docs/mdbook/src/installation.md000066400000000000000000000037401521727353300245240ustar00rootroot00000000000000 # Installation DataFusion is easy to install, just like any other Python library. ## Using uv If you do not yet have a virtual environment, create one: ```bash uv venv ``` You can add datafusion to your virtual environment with the usual: ```bash uv pip install datafusion ``` Or, to add to a project: ```bash uv add datafusion ``` ## Using pip ``` bash pip install datafusion ``` ## uv & JupyterLab setup This section explains how to install DataFusion in a uv environment with other libraries that allow for a nice Jupyter workflow. This setup is completely optional. These steps are only needed if you'd like to run DataFusion in a Jupyter notebook and have an interface like this: ![DataFusion in Jupyter](https://github.com/MrPowers/datafusion-book/raw/main/src/images/datafusion-jupyterlab.png) Create a virtual environment with DataFusion, Jupyter, and other useful dependencies and start the desktop application. ```bash uv venv uv pip install datafusion jupyterlab jupyterlab_code_formatter uv run jupyter lab ``` ## Examples See the [DataFusion Python Examples](https://github.com/apache/arrow-datafusion-python/tree/main/examples) for a variety of Python scripts that show DataFusion in action! apache-datafusion-python-54.0.0/docs/mdbook/src/quickstart.md000066400000000000000000000042641521727353300242170ustar00rootroot00000000000000 # DataFusion Quickstart You can easily query a DataFusion table with the Python API or with pure SQL. Let's create a small DataFrame and then run some queries with both APIs. Start by creating a DataFrame with four rows of data and two columns: `a` and `b`. ```python from datafusion import SessionContext ctx = SessionContext() df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]}, name="my_table") ``` Let's append a column to this DataFrame that adds columns `a` and `b` with the SQL API. ``` ctx.sql("select a, b, a + b as sum_a_b from my_table") +---+---+---------+ | a | b | sum_a_b | +---+---+---------+ | 1 | 4 | 5 | | 2 | 5 | 7 | | 3 | 6 | 9 | | 1 | 7 | 8 | +---+---+---------+ ``` DataFusion makes it easy to run SQL queries on DataFrames. Now let's run the same query with the DataFusion Python API: ```python from datafusion import col df.select( col("a"), col("b"), col("a") + col("b"), ) ``` We get the same result as before: ``` +---+---+-------------------------+ | a | b | my_table.a + my_table.b | +---+---+-------------------------+ | 1 | 4 | 5 | | 2 | 5 | 7 | | 3 | 6 | 9 | | 1 | 7 | 8 | +---+---+-------------------------+ ``` DataFusion also allows you to query data with a well-designed Python interface. Python users have two great ways to query DataFusion tables. apache-datafusion-python-54.0.0/docs/mdbook/src/usage/000077500000000000000000000000001521727353300226015ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/mdbook/src/usage/create-table.md000066400000000000000000000036701521727353300254610ustar00rootroot00000000000000 # DataFusion Create Table It's easy to create DataFusion tables from a variety of data sources. ## Create Table from Python Dictionary Here's how to create a DataFusion table from a Python dictionary: ```python from datafusion import SessionContext ctx = SessionContext() df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]}, name="my_table") ``` Supplying the `name` parameter is optional. You only need to name the table if you'd like to query it with the SQL API. You can also create a DataFrame without a name that can be queried with the Python API: ```python from datafusion import SessionContext ctx = SessionContext() df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]}) ``` ## Create Table from CSV You can read a CSV into a DataFusion DataFrame. Here's how to read the `G1_1e8_1e2_0_0.csv` file into a table named `csv_1e8`: ```python ctx.register_csv("csv_1e8", "G1_1e8_1e2_0_0.csv") ``` ## Create Table from Parquet You can read a Parquet file into a DataFusion DataFrame. Here's how to read the `yellow_tripdata_2021-01.parquet` file into a table named `taxi`. ```python ctx.register_table("taxi", "yellow_tripdata_2021-01.parquet") ``` apache-datafusion-python-54.0.0/docs/mdbook/src/usage/index.md000066400000000000000000000024441521727353300242360ustar00rootroot00000000000000 # Usage This section shows how to create DataFusion DataFrames from a variety of data sources like CSV files and Parquet files. You'll learn more about the SQL statements that are supported by DataFusion. You'll also learn about the DataFusion's Python API for querying data. The documentation will wrap up with a variety of real-world data processing tasks that are well suited for DataFusion. The lightning-fast speed and reliable execution makes DataFusion the best technology for a variety of data processing tasks. apache-datafusion-python-54.0.0/docs/mdbook/src/usage/query-plans.md000066400000000000000000000153271521727353300254130ustar00rootroot00000000000000 # DataFusion Query Plans DataFusion's `DataFrame` is a wrapper around a query plan. In this chapter we will learn how to view logical and physical query plans for DataFrames. ## Sample Data Let's go ahead and create a simple DataFrame. You can do this in the Python shell or in a notebook. ```python from datafusion import SessionContext ctx = SessionContext() df = ctx.from_pydict({"a": [1, 2, 3, 1], "b": [4, 5, 6, 7]}, name="my_table") ``` ## Logical Plan Next, let's look at the logical plan for this dataframe. ```python >>> df.logical_plan() TableScan: my_table ``` The logical plan here consists of a single `TableScan` operator. Let's make a more interesting plan by creating a new `DataFrame` representing an aggregate query with a filter. ```python >>> df = ctx.sql("SELECT a, sum(b) FROM my_table WHERE a < 3 GROUP BY a") ``` When we view the plan for this `DataFrame` we can see that there are now four operators in the plan, each representing a logical transformation of the data. We start with a `TableScan` to read the data, followed by a `Filter` to filter out rows that do not match the filter expression, then an `Aggregate` is performed. Finally, a `Projection` is applied to ensure that the order of the final columns matches the `SELECT` part of the SQL query. ```python >>> df.logical_plan() Projection: my_table.a, SUM(my_table.b) Aggregate: groupBy=[[my_table.a]], aggr=[[SUM(my_table.b)]] Filter: my_table.a < Int64(3) TableScan: my_table ``` ## Optimized Logical Plan DataFusion has a powerful query optimizer which will rewrite query plans to make them more efficient before they are executed. We can view the output of the optimized by viewint the optimized logical plan. ```python >>> df.optimized_logical_plan() Aggregate: groupBy=[[my_table.a]], aggr=[[SUM(my_table.b)]] Filter: my_table.a < Int64(3) TableScan: my_table projection=[a, b] ``` We can see that there are two key differences compared to the unoptimized logical plan: - The `Projection` has been removed because it was redundant in this case (the output of the `Aggregatge` plan already had the columns in the correct order). - The `TableScan` now has a projection pushed down so that it only reads the columns required to be able to execute the query. In this case the table only has two columns and we referenced them both in the query, but this optimization can be very effective in real-world queries against large tables. ## Physical Plan Logical plans provide a representation of "what" the query should do it. Physical plans explain "how" the query should be executed. We can view the physical plan (also known as an execution plan) using the `execution_plan` method. ```python >>> df.execution_plan() AggregateExec: mode=FinalPartitioned, gby=[a@0 as a], aggr=[SUM(my_table.b)] CoalesceBatchesExec: target_batch_size=8192 RepartitionExec: partitioning=Hash([Column { name: "a", index: 0 }], 48), input_partitions=48 AggregateExec: mode=Partial, gby=[a@0 as a], aggr=[SUM(my_table.b)] CoalesceBatchesExec: target_batch_size=8192 FilterExec: a@0 < 3 RepartitionExec: partitioning=RoundRobinBatch(48), input_partitions=1 MemoryExec: partitions=1, partition_sizes=[1] ``` The `TableScan` has now been replaced by a more specific `MemoryExec` for scanning the in-memory data. If we were querying a CSV file on disk then we would expect to see a `CsvExec` instead. This plan has additional operators that were not in the logical plan: - `RepartionExec` has been added so that the data can be split into partitions and processed in parallel using multiple cores. - `CoalesceBatchesExec` will combine small batches into larger batches to ensure that processing remains efficient. The `Aggregate` operator now appears twice. This is because aggregates are performed in a two step process. Data is aggregated within each partition in parallel and then those results (which could contain duplicate grouping keys) are combined and the aggregate operations is applied again. ## Creating Query Plan Diagrams DataFusion supports generating query plan diagrams in [DOT format](). DOT is a language for describing graphs and there are open source tools such as GraphViz that can render diagrams from DOT files. We can use the following code to generate a DOT file for a logical query plan. ```python >>> diagram = df.logical_plan().display_graphviz() >>> with open('plan.dot', 'w') as f: >>> f.write(diagram) ``` If we view the view, we will see the following content. ``` // Begin DataFusion GraphViz Plan (see https://graphviz.org) digraph { subgraph cluster_1 { graph[label="LogicalPlan"] 2[shape=box label="Projection: my_table.a, SUM(my_table.b)"] 3[shape=box label="Aggregate: groupBy=[[my_table.a]], aggr=[[SUM(my_table.b)]]"] 2 -> 3 [arrowhead=none, arrowtail=normal, dir=back] 4[shape=box label="Filter: my_table.a < Int64(3)"] 3 -> 4 [arrowhead=none, arrowtail=normal, dir=back] 5[shape=box label="TableScan: my_table"] 4 -> 5 [arrowhead=none, arrowtail=normal, dir=back] } subgraph cluster_6 { graph[label="Detailed LogicalPlan"] 7[shape=box label="Projection: my_table.a, SUM(my_table.b)\nSchema: [a:Int64;N, SUM(my_table.b):Int64;N]"] 8[shape=box label="Aggregate: groupBy=[[my_table.a]], aggr=[[SUM(my_table.b)]]\nSchema: [a:Int64;N, SUM(my_table.b):Int64;N]"] 7 -> 8 [arrowhead=none, arrowtail=normal, dir=back] 9[shape=box label="Filter: my_table.a < Int64(3)\nSchema: [a:Int64;N, b:Int64;N]"] 8 -> 9 [arrowhead=none, arrowtail=normal, dir=back] 10[shape=box label="TableScan: my_table\nSchema: [a:Int64;N, b:Int64;N]"] 9 -> 10 [arrowhead=none, arrowtail=normal, dir=back] } } // End DataFusion GraphViz Plan ``` We can use GraphViz from the command-line to convert this DOT file into an image. ```bash dot -Tsvg plan.dot > plan.svg ``` This generates the following diagram: ![Query Plan Diagram](../images/plan.svg) apache-datafusion-python-54.0.0/docs/mdbook/src/usage/query-table.md000066400000000000000000000061461521727353300253640ustar00rootroot00000000000000 # DataFusion Query Table DataFusion tables can be queried with SQL or with the Python API. Let's create a small table and show the different types of queries that can be run. ```python df = ctx.from_pydict( { "first_name": ["li", "wang", "ron", "amanda"], "age": [25, 75, 68, 18], "country": ["china", "china", "us", "us"], }, name="some_people", ) ``` Here's the data in the table: ``` +------------+-----+---------+ | first_name | age | country | +------------+-----+---------+ | li | 25 | china | | wang | 75 | china | | ron | 68 | us | | amanda | 18 | us | +------------+-----+---------+ ``` ## DataFusion Filter DataFrame Here's how to find all individuals that are older than 65 years old in the data with SQL: ``` ctx.sql("select * from some_people where age > 65") +------------+-----+---------+ | first_name | age | country | +------------+-----+---------+ | wang | 75 | china | | ron | 68 | us | +------------+-----+---------+ ``` Here's how to run the same query with Python: ```python df.filter(col("age") > lit(65)) ``` ``` +------------+-----+---------+ | first_name | age | country | +------------+-----+---------+ | wang | 75 | china | | ron | 68 | us | +------------+-----+---------+ ``` ## DataFusion Select Columns from DataFrame Here's how to select the `first_name` and `country` columns from the DataFrame with SQL: ``` ctx.sql("select first_name, country from some_people") +------------+---------+ | first_name | country | +------------+---------+ | li | china | | wang | china | | ron | us | | amanda | us | +------------+---------+ ``` Here's how to run the same query with Python: ```python df.select(col("first_name"), col("country")) ``` ``` +------------+---------+ | first_name | country | +------------+---------+ | li | china | | wang | china | | ron | us | | amanda | us | +------------+---------+ ``` ## DataFusion Aggregation Query Here's how to run a group by aggregation query: ``` ctx.sql("select country, count(*) as num_people from some_people group by country") +---------+------------+ | country | num_people | +---------+------------+ | china | 2 | | us | 2 | +---------+------------+ ``` apache-datafusion-python-54.0.0/docs/source/000077500000000000000000000000001521727353300207335ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/source/_static/000077500000000000000000000000001521727353300223615ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/source/_static/favicon.svg000066400000000000000000000102211521727353300245230ustar00rootroot00000000000000 apache-datafusion-python-54.0.0/docs/source/_static/images/000077500000000000000000000000001521727353300236265ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/source/_static/images/2x_bgwhite_original.png000066400000000000000000001660431521727353300302740ustar00rootroot00000000000000PNG  IHDRBPb pHYs%%IR$sRGBgAMA aIDATx ՝;=`Ǡ&DDeW\ĈFDUTo,"jv+"nD nd4YD@CACf_SuNUttw=IڮgU} iAA'ӮuN!ۻ !&Ǒhl!B1HQ:'hj!!TnW‡?EZ++E`7Ƃu [AGQI$BM!Fhŏܮ(%|a\YqDWoZg{[!Wm)lG̀(`@0>Bࡴ Y"]DֹJC ÷toL8i8FtFMn)gHDR R(+[VF(l O$J{)o2cP.s^@jd_`5xp`HrJ7[|sĐ  vƩ#O/,'bw BHax/*ʉEA,a\ute|Y4gQz JIŏТH6;F9w7C++I7! b쿯q:;CB! pcGWw|O<'ͳ{?BBgm[Gݛr~&Ž!so qD| $nā +eFW@(AckŞ ^A9C @eI7p6 j>sӅ4"RE@(AdZm1O +(}l\ E| +5/! FNQzOPÙhk^C_\ D/, BHLa)߱qu%^ hbf)]@XhO _ ^XC[x t #Jyc> ohʿD ,jJVL#)Gvc;A,U\-wu:AKh8g%e.bb꤁!$?tP&V>̮oW00@5;"Ȣ:A ҧ-̉lWb[m/S; @qw9Idha.Kp A@Ez"ʜt9bZY֥(g!:km%D~H0FbH%^NkObɒBn&BHDn~:=\$} ٓs-H8AC }ݔ]*Z? Dւ W RYe{T?dXr9%u&< ~@2uǖ>Tt3K߫PW7A)K)}lW𰅑ϣ/b15$ !O0JЋ(! s0@id,1$u@>v_N D>IHuJҝRlJ'4̼"PM-{#6'L]+ !qW @&S@N@𕕖(2$M$BHDx/\UXB­@ s/%0rX \n]}yɷ @T$ur{A4mLP%;1+a7%Y-FG9q}uxdD!!ÑXWW^;S BPP,q㛬7\D6!}"J6E-+DGY@2%ID?uW4#(ٔ>\Ƿ95D] @ .ʾ$9W ] ,I0s1*B3} @ {"a"U!^MD$Vx̔y^7(6Dd@^$cs)|efG#.SQN'N")}(N!$ =!\O]![6 4=őX_@mp_HS}AzEC!$p+U]'MGϜ%AQEq˶#n- v l}R ]#|YYjKy:x>YH; @RczW_m*]ןkqtۧ@ _XL 2_Xn/8Uf"IZ'!Cf^|(ߥ:'\ #MZF_ǑX3%q@p̉(uAtӕ+a*꙽{h]鵝՝Zo?l:OOFDOF@%RJI&x.ש:]]YPg­Mˆ 4ʇL_7N}Ru0 J3;z0p駜n$o[= _ N>h#_u?\Oyl}F 2Ks孷;ԩX8@=~fZT!5 rp k@q$Y7Sȩ‡]f_pD,C6jODM(fqvۯ]s7~J <~`O{P?HׯNI۲ ?!P6]r\y ,j}[( |bt o|m]tԈ ${z 6\$:Bço8"6>,Ã<,X ^:Z/~`m ( Yy*"Ã, M!5 I>0N(.NwzB N(HWhd AWQYĕryEAD\g$0<9zH~vWPd! < Wj ]:鿊|n #̣-do\! p<A "_4^x䖭?7 ,~)}kcO T T')`9 >TK?'0f]H GqEbHg6>ؚea X+_78,BHn{Re6&S-Ȃ$e_Nn6 Q'-r.8'pJ 0G_!z{Zbhzk "GQ_2tQ@<RET7'W 1c$ %>2NC ڰ+$uDW[^hRbh!v좨B!?^p ฎ\8 Fp7YpĄKei\!;6qa(M.7:Z!;/*%~:~7HP]!~aNoQg_ffGNkV(M5#uvݴǏ)Ab}j[> _[uׅ4l7!XB~N, 7H,q]!,|Wvj8{%e62Y2zBг1xP N'~]*5ΥIw ϧ:a\!m+pwӵT ZǷ:@nz.s{s8n®kJol&jj!vu4b/ZmRCmjl1tTGW\:V H#5ʍ.l|r4X(WPΏ\ČDŽvH?F ďB:(N[);\i}Η0aue|+צ~ڃbl v$ qI8xì "D յGہ f#j"Y\N'F>rz<#tZ@JG%y=J`u˗Y .}M.M7!D l?HUHП!!uu#QB|rmNfQo?\AYޜ}&ʮu^c$/ L j[Jc(V9 QC ,xβ|4l%9<1Maؑx{c|w<;Mm z_$a#ڎ'jiS" _xг &Ih*<`̅j*>Fϼ7kvOx~]Hxvj+n!I M;ҎR3&h6c?2os?%䬓Ԛ0FN!SZ!\Ju[K+PV{03Fp<֗s4!R;-x`KKV?nvՔQl#,F/. RbG4ҳmÅ< wB CݒiZ`t 9v AċH7X-}!7zM^ߦN=lk4`8r#X7:iW JIÒ׹Qd >Wo/rD[B8ܼLGZ1=|8F.sܗX.m(2q%L(0gqɌp#Ӽ4*pfWm$>dy7 }a{kqUߡr2vb7N9;\u>Ά4SE]v)[vyED$~ق^ɷv;PJYNID%X"+e@P+vî@@y~W#qs\D4.8ƒ+M1;9"T!p[ǻi )l"o2q4ՙ-O8Aø}ѧOݤE򯷫;@VA*H(0bGp;Gh'ⴢr}Y9\g"7&2m3J>u7ajR(NfD{XTi̊8PNIz*!8"I3y=.:#!,N?bnqZjSL JM@|H`}&PU޲.ϰYK8wzT[]1KgR'΍@XNSεoJ=fKY#vH{EDK("AW6PL }ÉOΰc®-2d_^G ! !&&ͦuR%䶭!餂H:FkPOY)"Jace!ܛP !pOF9Eד?ΨA4(Mj3H끷(YaI UBw:k ;].AUj5:7td| S5QO5"-a[iwJ-CdD'#BILٗ7(񀂏/ShK)k(;l nw|e".QcVDKLx])`_gq-Y>oc1D2@" ::x<ԀO-YCizAW,o&{4ISNi:n(zpO7@maygh`d#@z{^5\o(z@5}4j<{1ܮ" yY!'湱kD9F6[(߱ lj;xć(3Jl r/ܯvL*Tiv}h8>J!XE܎ -EDJsǔ (׾F"fW0LLr?vJF9 XRyڥ*{r:Jn׳fZ;I/Fs0UB9{T$Pȱэ^Fw*ڙ^pc#7W^K zs9-gǘZc'e-AEvdVZgo@Y.=6!sjM8kY!&F1cgغz]{uU!D.̌$,ןRu$y#N!^B G0\tG9z&=fW~Mylda sM%m;aC9c%M᎐R9PDBXvش|CZ)iK+@4ٟj<#.3E:k̩(-Kpf("։|.r@.KGLTKDu=~j {vsH+ueNODeV k'a|7'R "/[#;DwH˲%eC^Kij^ :xT' PNp@bi)#mĕ)tB8 kKչ=t{H$¹,;X:z9e>a`8r/ovnv( B΢FDs/%i7i}Z6^{ ԳT{r[)'HFK"9%VW ] !t\!1B4J9y9ѳGw[B8Ƚr?N"j@J-)   3 &.J84!d4FR_7v6D= TDmF~9Ѿ.7zksD0+HJЇ#sw :"3MG4N>׍n@_Oܬ$ڟ4wpT7ҟJiKHM"$ܫ(}"\ɔxX Dz"'.$2_zCjHɝ eGzR}%&9b!_[Qҿ!} ve :{e 4$N"\$c >+̲̉u Кv5-׃tΕDyXh"j^FBѤ?ǿkɷ)4P aJCBeI٧NJ@u)gxTXRt2߫ҝMnN/}}F%=|Y^VWNNmE ̥7EH1|+DGޱ>:ғ}4#2Dq׏")wdq'=e_g&oTqmSfw,B3xn"ސ@ Q`y)}"eQ,*$ˢI9Iz @T"*JK~-^j>5ձsh8KJޥ-Ё Ac{S;(65=EE4^'D\]9\nGhM-:.΋۱1xbF1JB/, MARrύ?3ថV4= ;v'.K!wD`NB +N<Iwbvc )xb# {uByjxJMMcg(!$=,ub쿯G7Id2S9Bn:v6sw p<0W֐ًSTq֫}C2g}~zcj,i;.1h֋Kږx&}Rl/dI Tj/G%A4K7_T7h7,!$]{ʢ sE@cA#\ Q'/1i @ϯ懔CVkNƩ-a YТHW*+buαY>~.?9:q픦^JSp.K}XbHnuT] n)<K-Z謈",k;RQ+{UIpizB 3_ɟ,o߱,<(Q+t$ 7^ %Ca?{_Nct#Xs((d'P>r*5̼RGLI}b:e>!wl/Kt*6,e53Nl0G0J҅B[$I!u )&РDSQ3=,8J9I1xzkBH`]8 +Ku0+*M4rD=v]cq{ JB]T!AL@kGٝ ɽ;ݕG\;7]X(OR% rB[(3\J|RQ.ԉ"׍Nx| <{%,VPcT<֦;KNi:cMGIB!s{EHȌJYX<.U#umgkXx(gKj/jJ(w.FϿE" x:93qkcXB. , ^Y^lj],R&QgQgUL)%}v eWhP78`e ⱦPf !LsP^WC{?m}!QGkO261Rr5i !v2ąp:A݌FF8W4̧$1!, l>T ,e(zQn_5vjl5e*v4|i95_5v蘊bd}> tR\I5Ax $CPO'_ȉ1K 6dERK !hak,>h5[BI#s 5M2M!o_@QzuK. SnDZSkY)udx@CKWQ^p/o;SSQX R%-D@DK슡qKv-|䞗 t$1LI=!:,8kŐB;A "AJbFƍ{'% !fII!#=njrXLJxC1)sʹ4fá H?Dm1IGïoD&8X+$fJ*c'[SE"P>Qwh_ehH7emͫoZAX qٱ> !y'?\M!ay*GhD=WY\0/mݗAB*IWݗ=3.+/^Fhwn xmtmx!wg3yL¤>{ a"ysM!Pq$ԥ(ܘ$AԛuzyyA@>ͫn $B_!Pz@M^$kcQaG[beR5BT wkPH4zH !OLƅQRz-Te_@1DQ3 YWoppDVs rmg >dT8 3$zUn:;vyEׯN (BHxœA. G H^滬܋DX[BC!Կfeݽ^XyHM Jj^re "}P{#+6 '"B9xqiV )osD$=HJӗ]|QY۳qUC6vJ[ɑXCx9i9D:(ޞ^D+nnQՅ•%dNG GbLŏb(AWR ` e_~ҽNv[Bi8\J0ce_^\-LwDv^Ͽ>x01/Ҽ/ P*<8zI_(y[\p}boWi?FVhNvĂr>,_;B|5_9E?KR>=E^!fˋ^N?rhRVg5TRg~V-)WKB۷o:sR Gwm#0,I?C};sYx%~|$@ "SܯU7w-ڼ2,gfSؤDm;s ج®ΤܫNܕa1#ʽ2L`pxf< G,zqI,n>O_ ՞ERƧ7~0^ K,tD |ï1Njk_jOLNՠ.MGHg.i~rA"㠝 >j(ITIQQL}~JMJc}ALXiJjmp~I;f4o+hij!qIzO#>8aLs8e=--wTR` 3|L]<]} >^,>IbP!|vic@Q|/_RX _:#X(}#M x@gB\t?sn){[cH5f_H}?FJOP%H, tyJ723)$ GLyO{B,!Sm ; FHGsw썧H|Ãdzfɷxxgtǹ_TRëG+5E$wќ?"t~ɯc=џTzbguW~b!ϬwD ?8B-fN3P+ 4,g־-_j]HKK%Ic&G "Rp-ϩx H5OX,~,0"q=u^DvڷB`ckHi!0zXiԴs~$}w>e_C]!#|!LJcR #J[MqA9>bʚME!J9}-#4唦'Mԋ!(@UHzOT<Ķv)(fO]c&Iݹ4/ o qX_H2.~^;˲/%][х3{bRj1Ϥwv鮐MD(w*i;ץ饢“X8~[LF/ @4QB/^&c]!L,/)Ւ܎mdOA/,Idz~:4G}_y)=1XM{@ǝB2'Ơw=4-4zqYD{;)鈖W"fWw'`kKڎ#šA=Jӷ@}}Β{nf KӃ‚H-?Ok-`c/L N[#({A=1\5#}9]#մŔ۹'M?Ls#ƨK!+}]!k.R5uuQ.i@G:x\NWG+*KڎܗaTDC}UDlľ@ '.7{_4l,(uєdr;^^a 4, rEܮmw$[ի(s< aÂHㅫ( *K.¢Hۼ( Do۽!Hfum:Mזm9BD5ni;)pt޽ûZ[!r\Ѱ_M߻֒c&=,TEY}Tqv0pz4RoFVk*3,Ih5 #zAHU+G,L]!-H-#b\ FJ*Բ_ "c0ֱo_+-j4<_Vpr_?q/ <,syd,Ǣae7qzB֟ΥɋH4}˿N떹9k,4|FJ:GdwglV>>ҧ\OxKe y1x0O?9IL츁@%!orWhC:OBAĐR{MV(]Du^d#3|xsjZxф] sSy4$q͔El -${mcV=Dc1-u۝>GǗՍ"N>K0u| 5A !=v|A#]ID&u!I&6;I 6JpEF IԋKbRo˺醹:x:x"5̓bDrʰ;uqYN$z@d_Ł,BtR> ~5vYnG/ *'EOIw[ЅOP#?2&ٷsB[1J1{!Ĕ̌EJINkgs˝:h -Pd }_(9A̱FҬH7voAy"C!23< :|F%.C U7kU4V?%:i`!zʿV-Od4>j|bc#Htp `XBH1I"cJaL7 b”(d\rόtwyx!9̽!\MMͣar|R9m)'('4U9>LDw䋲ёF?5"ts} !,^U0 5ʾ'85Ϳ| EC%zP8;dWS,9I9H>uVRF+iPAK}>u! Հj }I6Pn6swIJAE!g&kEVƅ(aDNJvw\!,A,"R+J˂+q$?x**x23ϣ1׮SZJTt>蠾ͧ ۜJB&mEBx]!Ér+_Cҋ0EI h$^!ZI}t* /?[DM >3kOt;g'"Ҏ"*BÐ 㱜erfN%M栬M}Sl2zse_>z ^Uvd>w.#C/m(w:KMf .QoDuY9=z(a~ZX΅Y[I"Y\>s0)⏞)Yjcy !nAuƐCBԧgSゕ>btSv"u Q)h,w+$v FJ;1Ju)bHNp`[z@zN+)g:2 b }Pu޶Oi“'D5 FJ1UFQsqro|҆_g8B@)"(ժ3O更Vjdve%5 'iDbV@|!08)1D,w c.:Brީ}(tHwdZ;B !rO+)=R.+g."jlG%|YF$yBċϧqp_F[Mf9"7ZU8"bJ'QY _2ȷ;Abe N#>'qrP(!$m !nzx2iX_*cs6_ bXe#ϻM #P{XWRKN$#H[ XaPB8xD/X#(N=ݔ |]Ԫ")FNcSἕ:ho_twwn8=g+ BL$ r Mĉ܎ys'%.Jjgb 7G0:x*JO+Jg8.+iw1& Gcsa9ZBCcwy.8gTn&iG{u <ޝo#Q8+չ@Y:/Ύ(m%oW;D"0B;BA5B:Y|yMw-d-K)K]F*uo@XEg i(K;)5#īb}vKpB,e,7`SZsɞ.+}߿MtG9رX8!)* 6Wز^Fro| U?HȋBE0)=!8,s࿈0%]{3~Xr_i)oxEx«}bA@+9o!eu.!nk;դ (O9!7 svvKEJ2uA^as"uP-R"[ܸ6@}/"i"q#k:b]=3O2`8\!$>CN  _}7.{>#sԂmtwZz~iRN )X\ uSe*Qb~ AK(r4{!M, R/_3iʿᬱPW/SS7K]J_5q]"I|,~lݔ:ʍer[PjcM@lRi$Z&XB !b\ tdA< Q{$~bqifG;f ok"Z1v<P X-M⠉pJ:}Xf-!$qEeaʐq%l+6jQxRH2BOz7h1`cjcAm4;)s]"p4}e@8B@uwŐA;B^@rQdKw{=[g4tB4|BHß.ϋ%ZwwS.X߽) 1AR╛=7D]6 :B@"<ŐFK 5onxdY$Ő/^9vBQcOjCQ:4|BHȣI+wrE*4CP}?E=/gY< 2;I-u>ؘjA@PhjoT!q <=1$ٮΥp' 8gRI%B'$sT'=:'8f/馰ᘫ/ܯv9^DcoYS(yGo"\û,X,PMh,#q9m)e/crD $a&JbwvrRAP*ӼQ@1)Bc!)LX٧Dcb2=\w砅ZEQY8!ƹC9!ΐ!yvjiюm4\BN>i5BB~^W%Wre>(!{Ϝ]/ J`Rw!XVn !m^Lr4,PW6L]!1XY2Ei M8|DP֥R1Sӏr4(Dj1|!9Cx3Du=CC|P>"~驋FBȬjuaDrr %숩+Da]ga7uEg@Y2g|phlqDF[ tH)&o!Q`7H ^-I1c #[^v3PW|z,$枧 ABy[5@(G#Ľnvk!\c4v<S/ )6h|{A eF2{WD-Dk8,=Ri;B3o)JĠ/,:RE`1$}Rdc,1I#!nԱ*e"pCKń)`h;vޮnzwskmu]Fb[ᇵOM󹹿uow{޷y1E-~u +@L+W,LRup7ܻPƿO">K OM1\Ma_`f3Tv=X{ b{@mtOy4Izlu!~zZHFBnõ\ :CY7A-/Noi t۵Dc&Yć'LS(Lf]-5lx^}?Lx0[hkQRjwZc?#8*hGG;?VKbYZw\!FP=/\@+Sp% !QZwhu'}{5"?]u!((*b\kE8qZ]̪gcZ!ƌ Uvs^S.GQÂU$ZkY 1D=i)8ΖR[l]7G}X/UL\Hڱt:Sw`8ET *]}R,NsYïʻW%A"mucEbd!Q]!:52Qt2P}kgpFG%W%z go*L(@hlq)K/Fԥ$Mv{E (p]z* <:nvt3]gˆ? #w%6b,'ڜsڿH1a 8lZ߱:댱^W#i%ͮ2ylWtg˞n5km!%iӂzx,vϠX4斪!ϵ;_K g{Y{.o AJ/v|n0zpl·f( MMR=!?3j5%mX9DFVu*~ vE,w8C#fBQ HqP"cBHqY%v_x j9ҟF7vU$΀iϧKHi v[zMbl P}Xv4̽Cn+(3sZۺ]>ޥ%:"\3"nPAD(ҺpQ_ ze_֞x7@=Iy`8AC,g7JeADVmlr !u${Y8o7eEci|eMb 5DȉI n˜)nZ Դ")w0/ӝ!ΪȢF"HSgh.SW+Pn]\U 6RQLT]Za;uGcIz%*y1u:V C K}[sr>D"qY#&L5L:)U@) Rtȿu-[H=^:h./0k@LރIYi_5 Φ̉yLB.Rmiٗ6*dμ%=㌯ ;#oSUT P3"!iE 0A+_?!'Vh`XFOHgr]qQȓw {ud4lNJ+p:">L00g@[^֏~H>6> APp;A)ot-WڥWP!P+<+$Y8b!Dk./t_vCu s-4/RGLoU4Hs^{PCTGP|a%Xb_p@qYJz;DbQs1D'Uz"jB;qN(,Pŀ eK9^ K+CX'.]~C uÌXaG}E :iGxp/SSi˾$ xTvi 9BL KiJJiw4Iz/OG$ Z9,T_أ! g0^rhC9G OPjW#D֤{ Skt,<,؁N0cJkl7 tW8bgO?X q3KI{|>|x?F"IQbB3אx@X8EU+)WZW  jzA,1ޥ}Yi 'Ϥ1 8_H!8͖?GY3H,<(P;9>(PY |0MDdv\weRE !C&xY ; $W~ S}lҋqYŽ# C=-k(W bʩ\gv X )' Aoq⒏#\! %rCt zAFޅ$`=Ȋ' F,VTa܈a)0StJ: wTkސ @6,,! ˖Fń@[k$R IW1$W(qQ@\/ D @DVjwwI@0U;*Gxɾn~MʠWn>>-pd1FEĠ$=ʚA±<p A.1d "+>X!(×]! r_VgV)v. &j!W\?+GX;K@Y !8.^ydW'd ؂ޮ$ƽG( n 22.ׯXIw"?XȪ*._E߿p:B)$BeE nwL4xIz.iUO}wi/y\6\ eӷIŐ>wvB\8DL .ĭ-* ]&%.gH ":y*o-oʹ UE=_Q \zH7E>A?!: R`v/nT5Pl 嶭su*8G^{*|rq$bH4kTO8br, 8w'Mu։%  $ '|b:]]oݩWwnŐ~+M>hߡ9`>61*o\#EQXAR-)o$jH!iN,)Y,g1eTIخ \v2>OŰ^Z=6Ǭ=z So>/=?ʽxa_@~w/F=!Nfѥ\@\SZIOPiŤ?M} oJ|`Q䭷߱78N0y>txBOk`*&Shn 4ֻh̙e>/ ɿv#z|`9Q~|2p;vSt]P^̻S's%$NfMrD jJ º_7_ió̊cIzKsvkâuPNt숬{w =T#_Rt}_/!qkTU[1eɈשYO}?FK6wֻ(vx#:FN4W&Kt#r82'.ԑ3(}-Mw"H:E[C$dѢ4 uxP4*5w+/r%6/?zevvz߀}=}7) },jwܚM6}Ǥƞw ~۔ H&BzXENcmS "e!cvdHtv/GATz9z׮b޷6dCrT.;@-h,Wh?JYk =D\a\$c>iZ9Cd(:nQA( U6 `F@i: p@u˗ZYy@gs; 1|]Ep  уʣ ?~TcMqD8 %Yɨb!D->o|f5(c.`99y"ޒŐgFo$-U, <:S>;1c_K_8.GfYkܕȾ΍NQ)k-U>1y:՚J}$H[U#-| $j?RKSnH,%2tm,z 'EQXRe%wBʁXAd2q= <,`51gп{ݭԽtrhJ.!c ֙JJ1RG o bGj^3A<Ó-Fba} F))Ko "Ȝ'23ҀbZ(u ?xO\/vINJ5~#{ f7%J>q"7 \Q+w$zD;@vL?CUz-+U6t8Vl#cbNU4qX/)шq-4n]<]{q]v Iwz =sRʿY i0bGa`G ?Dw_/rtYp>w )(=u Eٔ:tv*{b9,YB='e N0in3@2x>oYPKQ3ÏGp޻ʯvYC:Z!,}4b KSNPdII\R/Xˆm>H"FK]wXXie=|޽̦nmYG$(}Ma1{wD+EBYɠl!dls5s?2& cYzC`J/9rav X\'K`v/N bd?^ک!YYRiK)u)hoܢM]߱,f8Y1E pN}Yb3.̤7;nNOF]wGBnSm^~l‘Ab VܥD _\^49" vA=Xb1  8B@-1C1l'Ȇ"NCq.Jct#spM2Ѩhj]!ᤅ;= DwDwWHFi:2XBi:0:XJu˗%d8/\@QA "+% pgQ?Cn%{֜.]!fVa-pP1Ybܸ_/o-0Z Cng|9^agv0ɢn')d@ҖR s ߥ!d s/#*vpr+jJ=(M;Ie'*ߋϟ)A(_@ "+ޔ,p/H#0[D^D"}$kvsָĐt3BLbΐ!9tPu;f1GAtXZ+!i8:2_J]!SGwG !?RQYquZdp֟ ">t֡4Pg>>H{1{uW~otm$y5T|DdŗlYe㧍d00j1j9d J )k ϿIXH7Yʩ'Pb!zi\`fYA9.+^~/uDjTiR6X!ʾ.B@(!qtV{KрfE'9;,TL݋|#!`~cG7_CQW@`d "+$dDw fEs4EJ q)Cn{/Hb"U)$"/>^Uf/ΎtR%߳[]՛Bo !mćڔb_Hyrzye\!jDI [8?u``RXD0>7<0#V?A\' XĪr5HDd$9Ȋ'% !}u j@7bHӗxb⎠0!|9E߸rĸ^sHXC6Ax9n'PMd_wC^,A9?6Ό@9we/)~$:t1h} ?_%P%ؖΐ8b-Ď@Y3a),D  lf}T ‘AEKvo' 2Qс1mCg~2;h1{BsJT;o3f˸c޼;>.$}05gHw,v Fq\ @ UWaRMԡS+iQW[B[2Sk/L?13|P Bu=7φzb64 l BwP* _IMpš&h{G"ǰBFÔX||}I/ʐn-!d(rV ւb;6ەN-xޮnzލ rAbhn I(Njcb~q Qt2Rk*0 -ڋb\c=W!^2Z7>/+^E&)L[=JA4xjwHt aßsYq 9KG܂B$ 8F"ÐBH}=SK9x""#PzN$n'I2ZUGA6k0)\a !|9q( (nSă 2(S""[sMTKBׄcCF aŻ"!A&|[&8Q%Z"R(kw8,u#̂-[Q ]ج`MB,Vphj lAIï%b\I&"$5HbbHϕP@nm);CDh2T :=lj0Jdo7 ʿ|;ye|" ih{EK}6j(:r"2Yv"k=;GߡZ!N9s~ح+y6K."MŷFZ,^ ,d׆((`PkkxN#J+QCf1]jޗ} 170LWg6;{Vߏ Ѣ@ظގ*299f~g)L I:߾Ȋ>!Dbu-Q[Z(3%ŘeP_?b^SG^ /-soFDp#^`|_|镢{&Fa+t.XT&<pZ;vʻÉCS =:u+NM!$H_-\ "ZJ+ CG3}1JV_Nٟ1KQ2b ùd:?AFʍҍ='Y\Ly#5t4T޿ji$Y=t\ OHj J9 ^7RF<|o>OTgvx?^4a8F*hTu~Hg6K ,[ Gr>fq|astٻ7RAPÄb vӕW{*"fa80 䏆0UxI>\!$۾ur\A8s.[y?'CR*≂Ep36z (WSwx_`Iiߖ0n`$s"5Oϋ}IK&IkKx6ڰ;B6>lͿh\8a5o߱Qp %uu:4 † K$Z>%0r?b#"yytDn҇*v߂-^BiLE&mpa]J1+ x(;S\g$7۲I5X"uD;KGqLT. Ѕ03`1T0珮csC=!.kMAH;!#Բ㡫ȊcY !?T" Cq @c-1 x}0[GatQνAn9=|}ayO&eJ[6"ć&PQ()pi:D >!ʴ7arZa5(kZ"; Ő8aSbFw 6+wBhm oBI]mS2*&"E !dG;eUAZn2'E'~H,7De)Էr_NYҗs\D௜xtm$&Rc!C$v;zB@U S 0fT:}d323P *ʻWQXy$qma“;] a,8Db0_Cђ:@ϯ+"c !wð!Dcq-49c^ɇ * AYx1g zCmwʽܜޮ&X y'Rć(itZW&av P5a,~f1@w1PI++YBt,$f1$Ֆ܌ƿOo{at c7]{ ,~Cn_B=F#>:Mz GIi㙗QìK)Ɉ/rk쪁@EYTkF[^p{!5sWa㵏XJl0>1Ya~7<ǂY@ !mFƷxhMw8x#~>Ì2VL/"'KXB/skh2J2q e]pJOGIG4[v3J ]=!F  Z³*0̘:D Pc{w 3ғF "jOҙ`AA &D}8<{ =Fg gBz.AzQx%.|ߥC# 6V2TDj\RA%Ǧۉt*zfFpe(M#eˋ (,Ti:bEB.H57_"jKJr3"+|>K\v:ݱmYAhC:Un!pē懩w]. !*ȸ,},a@V$ee ,EW1^_p$ cұX˚QFkYՀ0`тrRݸ±sY@@eKP <ׯXI6rr O8.>g4[UT)FXY wH\] SNR +>yYo'g!==uvrKчu/|=! yԺxz4?jF]ּUؾc:>yx|  !Xd {]b!F ڐw,v7GfzDTPRT,\!AoQϳ;d)!Q'Ŕ{%e#`rߏ Ʊ~6- "JDc*QO?p,>a5%j4a JQECR ş0z8=*Y߼LqxYz:p.&pg;YK v+E;Q &PkN;Հd\,_-dlNztt&>!0jgdč0:-ሺ0ÍOCDC@ 7x@(Ed} TuYH!#"jiH|~̓E.$ʕ;DBx_Fo%v!7<˂p./g@im>}Thqd/Auػ>Du'$=ʚA\ÕUpsC=xm`$粎˪wAd](L  a;pЅhD"K*eJr'q-,=PPn p"ع֠L-J_!=Q?܈.Aw_Fwx\B H1Y;񧎘D>jo|b +hP=BuZ0E{c$geP v-qX*YbHX:8B5w׽ 1aN@xLT {Rϟ7zǤ WU^(~zHzB|3େܾ2J) B v=~) Xzd7=L7G=Zv.CuH:fc[9RӒHЦ;v]!u<#((tXӹGEny1yAj)e`8B9%浾?1 W nzGȬw`o9X)z+C-MxC$*9.@yL>6Q5#'TzEd]j !zt?3+&x\d08$!+>M}9REcb80/)CfWE@ńQj2΢KQQX&7Qۋ̪t[NXp!Ru~aa54 `j3! K$lk-gx!dtt'{QȪ<!9G7 vjo0pב'vn&+Qx*S&e'hهe{"U4CPF@4PKqޕc_9:~At,a2{ T966ay8 .kFi:u'$=H?*+Ա;DDt |*0C_ OHh, f "*B|qXq^Fק:bO%==Jo[#Qlw7½OkUd|R`GQn&= !s!e>X#Re)s4{{n"]Ev.$*E3&[/a| kFxe(MY^BX}|aqC@бYaQ$pG!Y򤲸n]aW? ʒe.N,;ruZ>zd;%ftVDUj·6P>i=u_6 GϨᏃb{nt~+3ƹKF[XNGጂԂτ7s3x@qz9bABϬ 9vӛB$A3䁹f;rizTrD9pEE4Aa>%Ma q!lg[&H%\>+ Cuh8"'Cbd`)0ٗU%âa+tc2ma^iX#츑=]E!<1W7ҥ_a!ńFe㍔zjWS=Pro*Knf1u񌒍?DDG7 5Vwo|y 8x|ZyW8z5縎zσ|HZ+ ;l14eV6[ Ѫ|k2QE9.oYIQ#?r?ёJ2RXĸOpcy kv~ %iws;m}drwݸz񦺋R= _J;,.] :|mq_mu[Y"ݾx\B0N)ZagIaXök.MGY3Є1 4p? x@+wǭ*% t=j(TOȎL) %,>"F*f !s <_?2wx\z)啀رK iZs!s-@bUj`ԽlՁ "-у]j bf7tH &3SWSȩTwA.~oIv{@8H1{) pc\EY3ЄLvi:ψzECXh%ahUk}pE%|=-5nzόZ>xupGHDbFd- lD>[B C[fҳpiE/{*b=A?D ϣS80924"\oē{b]-9.bp?Hn $&o4V܍džo+G(ajZy?5 J65Sz, 8ϖ( q: .vL?ߧ{!D>&0]AeɘifݻT<w^駐ew0V"^ Y{{Q4.x?0\:ΖiЏZ]M瞤ǾP] bPd5Ђ1i'h~j  'k}wrc-qVdk|l;̧1KD"7y_ߩ`Lgʑ{7QYH1EQΣmD*j!PYo%5665omN5z"OIY ro'Ybm:#P9Ŧyc%ٖ:g Q9܈w;3}ycvplj' A/WI#6':P`!H/HCa>A 1IuM/,Njk0wD30o f7Ymzސ!pu%}]jyԿuF"}}!% Ԙ˨N8o޵'&XL3SWA刂յQ-ʞљ}J DGL Pk,qփF`/ ."59JswA@3TQ+E@!ig1y-|# b E9F_=ܺDb,dN8y2K"6G0⮑$ҝe}0<}!d7e j;Ty8D֮[GJ# G ' w((:@? '`$Hb WA8B*Db҉l{[hA7}X4P=\GV " cn;WR0b0+A4nt\5v}朑BMY=1b7`!-&֯`*C"hmSŠ98jr$=r4 FXcl9%0nU;{n9*((c v0QCU4{I G F(?"9S ʚDE|Fb5N/~>aoVOpLNޤRQwC :'0r30 "#(WiĸJvב }Z(q纬%}#p6Zn 5Zˆp 9uE3GA1Bt@~xu[HߪDrY/NZ/QX41Y]xN&|<sÔ9<N?gvro' D^6Beyb,rGnlBLPDXJ,Q$#‚Gn}WK?AQļ34Mg°||$>?ٟ2:k(*@g>5Dtx@ai_@2 ʚ')Ϡr~ءLBwM"0!!*Qf,R\bedD WDd Xų-|Izg'\w#V ,IW-3~6N匨 ͡D.\E /pN)J>E.tS[h..?Ty)P Ks+ iaRS@M_Z )b^gt]rڬ7VHIOM B9ga$3q/((kN.03z\ ϞschN8BjCX);wqI@mManj]!V {. oyKf>*’߸.վȲ|jʜ~Guc)]GXNX9k%rN3Hm`8.Bُ8Pm;2Jӓ˺16>t't[ݞw/(Ǔ/x<y$͌ ?l۪=f]$^=+%zL'd)+id1E^Rdho]@)$l._a4^sH,aQd辕ƭ':QVHp{ϕ4/:CD?̹u99+D/>y1"IgR$2ry~_^ڬ_iX*+GA-ix)!$ט-SlԺϿ2̚ |t`в߸!yy,m87:S޹ͫ v#2Be0(}5 -6B pј b6F6BW.yF,(/Y"F__bD::M{F iZD:EiV.rj2FB%.笠zJ'OR9{HvF-("1zO7CQ#E?DĐhSx$AnM5ˬ Z |lP%py,d4Uu _4r 8`e?)~mY9g&ڴ< "Wz`ZV7B^L1*j8+*-ߛel#u6Pb (X:kH_eSG6>!cQCK+>5!$));1ҿN4VYU 1AeH"@;.āb!Aɲs)o}V-"{t|F"|dqa9onSV8Rj jSN̎b+ ,fpIyDTɶG, ,,N^K;+YM;hy#Jdݴ@6)5R[-LDDT},~^ޚ@5Һt]s6=~UL+2T M\be=ayg_1#/NT"GUMr(T~F~|'Uئ> kW_R,tܗ8x _1{2fxz46PZ|-d1Pse88Hξ&%)cW~ޗ`,ݳ,Mtv %E><& ( &/F^z^cΑqYNGy[a`Zt|]Q . F,Ux7{B҇z4E{ȡIKGU¼/2{PM/nByY'inE"=!K@5~G47-Fħ)-,|/:x#x|%'uҊ1$8:=2{)Xɺal,jEd`{5c>,ɍ;Y+պf~Gy h$V6@lGB٪L֮GSeʞ Dv{H_Sk1M/Tʐ0ƅC-Hg?)lM\OQJeզ#pY7%J>&U keF6 o^թ  R1FRlo8aNA*OM<$ₗd Ƽѵ ^\  &zVI&k%(&[jtx}F-wOҾ92uy #gMe=h#>m{g*7$s) ^qřO`[uÝl8-w/ Ǣrؚ_MDIL53{,0m_[p"Pt xQۂE7茏'|ʿSގhFcty..w%17: H+ep@ 4ȹޤDCZ[onO~}q5C C=0^ܑ ’O"8p-@8~ʿ Mέ*Wdw i먵.Rک-{K_Y1L Lg+IEuuSH5jŸM3F{ڡH0bx{XTT~a-Uv@A&jW h7i2DN=g\¥ް|#f~aJ,LK7ZTL>i:0[VH|22[XX bΈbHgfXكE1lZa׃Ö:3$a ‰hWH:zʛ{iMj|qVw}uY+<we.q >/+|/oґv(*oz-/rW7uf9"epIlYH3H,c0060JY%gqG(IV8,mMC 'p7kY#0RY U(J;c?U^}F f/Xǝ/ӼLֿDyzxoi0+z1'(G/4zP2, 0M."8S'+ U[ L.v5s#<>i=}B aX q=B\ߐP$rY'Ha )atTtSQaCaeT8l=QD]:зԙ("#HJLJevŽ&bٍmgl+)2 kk5IS!{k]DRyyg׍<0IqZ5i{\RKP ,uɞ"wo'9ʞ !5ҥ?~õm\v p뮹 gOm`"͖7!$hz¿7Mq87Fe*?`V,jDo^`\wU%ߋg!bV&fzo+qI*?7l eG0m>'||Oi=s =fH=2Mi 0+բQ_啡3c+,<!ݴaFE B1? Xq3+eןYw<$\؞K8l){>~U'~@2qM8EYTǯ7Sˁ>$FGpӦU$r_ ,5CğtSo$q0Do֛_uPir<5=z,!߃|/USo},5gJbw o彑 78[Ilg.Eaz°lvΦ皤{RGy`Zs uuQqjꗦ 5 NUxT5(cpP W>NtvEBN&BXx9o @|5’Xʬ IwiX1i{9$M6sCp϶-Dxcl 7+D:}=P9Ƕ'N#Ťw{]y[ۻX9`(oy6r?FR ^pm$c*,zMeA2Du+!Y_!'G}g^3J"o(Z0G:A^a4M|v'@l * ;(Qq$LCSZ1 yFGCjQU+%azH["zt&?ďT<8gnU~G#/zր~)٪Lp? A1ɜXsxI|oMUřwiszC5JWfAgX+sUY>dX7NV} n=2*swҮÚF`A5H#cfô>.\Gif 6#[5Ps6/{@/|o-tT@&.u SnlCeA‰["p['+p[wƲQd4}( |TA{oOcci>6#5B`!dP98'0*f"lc,fP°|7G+m ciԱ.?_KE2 HKa!X2=Dz .g#`)<&mM ;Nv S@G?c(~Gu|F6?KT1h>c x̌sZd[ 3D)|ybe q0T bםt?,n;8HC12 F[R~w[! 7k^bz@<ـz>`'&0|Oe Xd~ߠW^J`bɚFb?tew5$`bPvԼ4V5X mMyQj9u S $j>!BŶfa͌(y(_ĆEX9an!vhm|wn:ZN_4\{5ic|ج9:ڬ9m@#@eX I7G{7um`oܠÁwh>1{)s@pm.ϲ1 !&lpɬF6GfIN[y;B$,ñg!bG*no_[ƶAbJ96ԏԡ1E #fQ>J\ILҬ9c0Raamp%HeL):˜"CbD(Jhl_N.f F$wF6QY1*`h <@ ()e%lϤR' YiY3tQu,D875!a!`<8 /||T-~?7'/8D5|mt2%h8Ku/(FOd`@fk$8L$s$l7RD*Ha]gz d(3 fQ,rc9 )~5@M~Wiyo<46l˸G7λ+ E]E`ɼbf01‚He[z|cF% B0Eǃ6Xg" Ff'9{:_bV*Uᬑյ|p|AlUY#j7(?0/UŰdٚCR?>s . Q|LX(ߴ7/FDYDg>i:>x.fF&wB gMux%x+F=*INr~V $s"_%%D ^@]tu)Fu6.=DCl;HAE #̱Ƴ¢#w6#HZ+XH2(#BɊP Ng{8өTDDO8`4(`9eߊP-1N?O& f0MO4LQ+|AT(SfW:2 =1=Dyy_o6}`,ό{n;5*BY'$" rp [cRRK(JZ4KxE! ӃЃRI`! N'NԼivӤkyb,]^70To]fF%PD %htx_d%zvo=S h|$EczC#=XTLNf0M?}@{u_ZŔY}%XY=OJ#|7w 0|L86%i¿)|,t]rG !{ J8ؙp4M(gΧzPeD2y^ҘyԹnadwO9kBO@- Ϡc?|QIA& fh^_>V럅Z4/* $Fe0hZ?G;,x眳DXi@X뚡|gVx^{ JdMM- E 7Fo@M+,04zc N'N#c jC6Cd>Of0M !\ 4~k;17Z0xB1D sp^тkxqLiYRJuaϦ~٭a{CЇ* M7"+$ svrA'oJ̧y-I}HK2G}w')S԰iե~\fd3ң-wU:8@/|h^yo׽>9o%1nF 7 Fb0;HO: qz}Y֯U q$d3D~kP $NɰR4Ϯo$#;a.5 |&B[O){D BR=>Μo Ҩ$QBt_~V"Pvv/J)aJ>!,[Hh"*A`f>h?쐡-Ԗu@b M Hז83룮m\d0Tc }$P*}4X?]M HvPѐA},-pHD+7i&wl['n"BHp,993 (eg%FƔ$0&o *Yx[$ց(YBHʰq["f gߨ[^!B ^}/7 L \7:5ဠSt@ցZ>9NPJI H8w Xr B&҈'0"0)I$BHF%0E}1͜/da_8i=% /@a, č@p@g #z~i,7:Zޣ$wY5Zo@P  %Z?@ sևF >0 QerƈᎅD#wn+۾ϊbQJ<!$c}q3<,AT⇪ˊ+Uʯ3ʿfO=:m]N ?@\"s-T BC;XFv Λ&f?* F -G-'G RtjC!LB\C!f,1 U%@ A*:QzƗ @Q+- abdm"C3Iy"!Pt2A;IM ~!PimkCuiʊ ٱR!c?/9BHa%$>?39,m$r*oC D (F˖3rS8ڿ'?#^!b*oN;6wNÏtJkjBo Q+2X/HR&H2CO ;'"‘<DzQ 7ٙs!TŐʯR3 ]ߺF7mG5F&w'(s )P::{P X'7Xʹ!"\&-Z?;/zih2dXp|DdX9:4 BrȮnօ9K? U!a"VH‚H1hLj>Aɱl:h .Te,CD":of"2>tfH4MM7ܥyKa^ʯFI q@c!$KLJzV_ nH6ag,(lxi]]<( и@1Z m?LVANlu r}C!)O[Ket6 Dhl Cvxt?B#stc`>_0s&>mOgG}D4C4aQʾOZ! ®7𓁁cgKQQƈb89 r"*ٓj+qʂbH4鲛A`b8l@K0'[g:'SsTɬ_ΠzQy/F}< @ sx㊞xÙ =@i@FHnQeN8ltS9 3ƍd)"WQ&OxFďR5 ?<&]~yҠT\Y LExyo-(R8l:fDijxo~*^HLSml;s4Bcv{9<;, "MJ¥_FKo2(]0'wROyJu%(Y)x5oR=9 ?7g@26OT &{P!Ni̩׭l"5tWŏI_@Ɂ$p]oM)A"2XIY NY zsZQ4;MoGZN=!M!caDgjDI֞#z9W4)IiM n1_,BAZee4\+sB*F2hEl9V_ީv"*͙G &F˺ea\fXyi dˤZOO-9Dd\ ~z0B1LF0Kg3g ! dt&%fY=+zy Xdhnz{CyV)7#b#B9OE򥙳=<*`d䅍4u埱(2ƹqmD]S"43=9?LB>Đ?S^'=aDIz ]J(4wM(y6 g# IENDB`apache-datafusion-python-54.0.0/docs/source/_static/images/original.png000066400000000000000000000633411521727353300261470ustar00rootroot00000000000000PNG  IHDR!~ pHYs  sRGBgAMA afvIDATx[pivyN .Dlc30aAGL;01!_M,h-kfytK2 u#kF(ڒ5^[K0lK !͞z`Rލy|ߗYYXUYw_7XYUYYYp] rFTJO= yO ާyėAAA8\̣{MZ0A7V+‰$  P/BX|xB t8)fBe%TH &},?K"FAA-BE0O|:bχqsxuI V;ԉ*KJAA)BAA 0eE Ϋ  a@{^L Bj!PG{AAsDוeS:Rz:YZ j܊_O4߀p-HI/WDAA8br?t `.*N9'<ȯq[ރVuSUZ,5HAAr"Dv1uNRuGR j 1BoN!4^$cz87p AA1" N :zCt V[(SkA=NûHǾ   bEȓ|y =o<4χ(6h#E1RUW  pZSһɴ#B{unxEl z~ﮀ   RQX^ocnR~ Z8]|o"9MQ<,HxAAa:•I^:n05ƊI'1R4"qu>3 AA!ƃ!*I +Wv" H3 BX L[C_iȫAAAbJlۋ$3fav1+Ϝ)~s  !>UAA!fhD/˪h!|;mgx 1B|   CPl:N_LJKBn^wЇl%N$ڙ/!  1rnHV9 +!XFXB+*@඙ɪAAA!"^3iZ0>R>fr! RR   C@c0\: Mr'~)`x`6-Ąe  C@{B"9<D0E(E0/|1\SW8 I  P1" $7 9l+Hdt/€1s\"djo_AAAV`4b5x^n9ȐK cQAAaa'o:c2W) Y VunE:i̝( > h'w#3iY\Yz 4nqG=W+v-~  db7"zm!bz E^973<"anN?﹥F;3li}уzU+R1 ;d /v%~-ܞ8b¥ߋ$45#ԸD]Emz|ږz}"t vB3SA=dro?ۙ(DXtpXn, AJvxĢ.S`C˯s e8,@OӵgGGy AAHX>F蘤4k) <=_Vr9ȗwMP4<"]G7a| &$(ȷ~n(K xࣺ&{ۯ]=ޝt k \#!PU^#< Htkm-1 S/Z_8No $ ]!S_y8Aw } yN0'FV?:qHɣ&7 :h`$Gg A~4$xMaNDkJa\LcMubʙ4#yxFgoѻ7H ?}zy4:n&huS ܻfYgio5Rke;'kR`yڑKP/uoWÚ/8G.A"62#D.L_Ep-s  ´kt OCR*ɽE0Gcbch>k'$9%V ] QۅsN@. qNz"ć.PۨGuD`թ~U4͎0O stAfEć  &^ֳnTݨsx7PGNm1'W1#pMlL6…#Oӟx+"jo Ht/SX6(y4@_( 2~(a#ywm@]\sѫ$\x)m^,ydp5omM# }Π%cЇQy+ da8faBoFʇD ʳH+ySoE<$ͳ~16G;eԅt3:ʑa=e.bd k9!1 ВLxl@r9( SFw3OM^obwaw찁l+80H:F TgZ6_K"~7AAӴyU!>0 2ZnVtԂX8Qa _XV3FT-{%SeGzM(!|-N^+k}v,6*Xx<%QyWi7yNFylFBշiEZ5}#(3x,o>y>9gMWsc>tjB쌳0hcc N@FQoKVgQ*I߆" O"!t&"o Ʈe.]<^ya{ B{hXl~nT\eج! T!=Y:qo~/kty!Z&8 <_ZW#A_?V pVwA@7kTy֔ǵރ-2CƃD職Ej.cIh_زtl{#:`i 2uJ|u?}:r*85{q*{,`duU8`otPzK"{/Za1ng 7Q8Cza{ BhH</ݢtޘNlE W| pCsm~n4 ,[1dw\Vӟ}GH=7K#ה.l6=(%иUՀ9зP#ARjPh{RYNp'ǔ OH?k[R.\Z?bQ^Q#nC[yLcU8]ISە>VIdAA"~P E7nf~c Dj,omb'ipCOC] aR$oyN ' ) }>)jHqͅE+Nvo9/nj>;5\Pd,&? 4計?uyJC?`ggV!ŒyAkx-$ȢRn7XD f^P,3ү5F_+AHd!p\[a8ipN2OGӞz{Ms} i4tTtRշ`[2t9A.A}@:5t9?X`1G$賣3AAGzBˋ t<97ڇ3^y]>|7Q?݁y"9`\vذE鑑r!../'Z%|<[y&mum=v-, ,T| [45q%c~ԉ<; L}wyZ?A'}|(hMC}/25Zp׏}6Ao>*&ֿ H.}Ǖ&X:ol5qfU. 1BtJ-zr9Nhp2 !T*vN(7L¬*ҾG^4+zf D ']7Sg8Xt\>C@A*DRu*߻۵k Ty,3 t F|̓"pnyZg5C&ߛAh=Bbjٱ"Boh90Dt;~r|h3ݪ4(NS3nX*oHS[%1PĈ  TH±6g9Y׷ .ߍr+]K.LqwxCx¦b̿Ux42>y("™_]`glHD2S8 _R]5ɍ  0gBK˰u%bsKG(TCAd4Jd7%/7uңG>: OMf|W9ǯ^4-¥G5^p=Aᤓ`Jl;!W>Ag[a#z|_2х_|LfĠ&QWp\jJGI'oJvF>7!csQi8B=-a<}N<:7^~ҫ›iIt^L99DI7b:27KAhE^Np^7NԁN#+m0GmW=O:uzӫ$@H|?~{kgdgޛɣGս_x-w8T3tk{x58&wTp 'ӾvhW^Qޓt6bQyWZes=4XtU#޴, <DJ4v6eY+ɨ6b>~b`>ht^SG-?^i=ڛ3[|^Ăp9|7B@[9i(GZsS2 h N'μ>\K נ;Θ y9F,@@ |AAh361]iv7&׺7z &/Rn)钞lzGޚ}vOix8ʷаLtX흡LנHڬd==S;i%x:i.n@C+gH^уB;tN_#"jb6Tr0+.aL֗]pxF4Z&%˶Sp俜΁   < tO@(51嘏SmHVBbiͫ2.H.g~?nL)~qD?ZoS;TM0貵  aʤ5+lIsq\xEiJųduw終Hw~ʽZ;NE#}WM5  P*OI >uM(]XdFisgC8 EOzRAKgt{.kO0AAAHCI^D@JU+>1t~1mójpvɅpk]}2Ge 7 4-rNN   pNf*]6 U8ĆQx8CQꞭ8I.sѩ  B 'U,TF5ֺ-vhO Ta*h}EOK"&tپ =!te    ?\NtMU%?nQ+3uJrիAK;2C@aAA!5sӮU~s3"\oC@*'FĘ=K.j۹|7Bvl芀GyѹTAAA`<}U%Tz/({t?pyFm>YJ'lddVj;]";iN'+xt.(  TMmlQAW߁Y|U"B~%6$;G 4 >{;uBAAAY!w¸bD SARۅidoo_>l$Wԇ{럮Q M!n!w1o+e[]9zl=)L5+d4W   5RָQ AM8،F]mxcLu^}&9g̃F!MIbaqgF(OKƤBqAA#Br;kY$ØnKH֓̔N>7`!{B|Y: y#Z{F70 5z?wƆ\<p2rkb. AAAj0"B<7j] K6J*Y3StAAh% *}Qoȓ5z[VjļNHG'KaBÔ_z'-+D \BxAnˤ  T?'aAy8|o .lB/@*1T.;#6>*!0v5 #+(  H Ըqώ]ސ0[6s@V%~1L1׫F+8ly`L:F`eU="FL|e䣃Hϛ  } WMqeMqOyhe[pН( ]@~O}^ؾx~;?zv^y+w6@J5c0aWJi@ϊ0;Mj߻9e6J~sN%$ZD~ o[>!Z&gT ̉@Az1' tkzn\SZcki`Zdti}p¥ P6<7?P@y fM1vdC?,':~YwYiG uύ.҂Ex&tf?W~>y-L^p>v",Z'ek|8BS]>},gYx:Tmҭ󽖌\7lϣUَ+=H0ú'2RwM;;01vKa q\) $Xiݗlhy`\C/yr#:BBFvG`Wy)ys4GU&~o >t?H$4ѳm80&wE4,߀e[3݁r-Bt :q ey/wq]ćȹlE+>xNc?.]C2.`\Ex\ @^C x}w>'@F_AF8$Bo5XzV2nlqnHeJ ]p69QѦz!/H bZl+V"@ZRMgBr5y޾aP4qa+T:fiWݠ Cb yP"d#oF-F*~Sr ^P;BE:m- 6{gv u6zwO-`-€A`Λ?`yyW= ! 熐:*!^|NﮒX86#@:4V JlB"6ď%:&VP(u: ׌RS]I ]<<}Bh1Tz Bg8} bG-FʥACm#8?:.nVͯ#Ң:~5GۼQv[m+_MbbG. oyrڏHDyRK h䉸nڌ$7w9 ҮB b & u "C5>>:QN6YR+ vQdJa=MkG_OM^: B[Mq97A̓wkXs-w; ~MVz׊0kp#Clg|hyz;fXMU SV2O˚/>yV[bR K#ڬ $M"oHc]w~늻#V{L SIo5=0[ (=@p59LrsmҚFvgTF 8#n({툈Y|n7*ףGE~nuS`H\fȅ` 68@4TEQ@^[)BA"]Cae߮#犐`Gui.KOW+0I I.ϣu"3 ExŀJP־I?{IDc$C±r0E#9\+C. {FHP./rׂez<Qh "ks|"T+psiנ}%%71ݼ.Ģ9V< tj[E|'i8PDpӴVCN'1~K. mayqv۸y21Lv7aYNJ~pԱGc>Ofry>8v)!%FKΎ:wџu3X.-/ L9vyA2{là^?{n[A"Lz$DB_,$*uUK~"x{smD1S)=twV\&+־zY"%U5*osx_B3߲/\6IevؤtNf/ f2Q0t27A8oŶF|=*oZ'Cdk^'1!O,'C}4kdA&v갬zkΈMW98tKv:OJb c\?zI<p2<#N7EF*i7)oBpۛfAhd7Ԩ3R0wZݏt Cs {ق1PC;r}{ .a:,B t i/Ny7*'!rˆ}CnE|Ah|qt'lT*-+mS)Kc2bSZ0Kc$#~.s^ {7]hInHFCC 1: dE5mr^Jbת ˪!0]MK4eƼw琌_N$*#x\Cs%D Cᦑs_~-SV( #9xP#GL#ܜ FfjȖc(db|ߺi4(vos#A-to7CU%BrQl4n|P_&Y%7b7Zb)J[q47Ky8@Wqqiϔn]06|+t+JYMbG!K=||Zx3w960p [:i7. t1Bzɘ=eA3aY)k%'R6dd-C)N;@_*o̷0-GbX?9к:'y%&_&RYNytoHFSfuqr7iZ9$7s [ 'a PȄ,aT*S_ /֟ t^ ]XV"BrX~5~PZ^/w"R+1iFHrAjEb iP ˓"-WܬՒiS{1vݴk 4k Yxscj=Y!1_Cǰe;TP8ad#W.(l >@ u+FFְ1x"]Α#D6#t%\M;\'RZBxBc}M2H6? #belb=n̍{m3­~厎Kj`#{ġ/P:CFxd@=ʿ0Np }Qe@r .X8ϼN.{҉8CwOБKgSP)˼{ 4 eeWhn|6';cc3,1Цt mV܇}[:Nh!e9!&tH+Ds%,TRTʨܩUn7uPQ$ 5}U0D[e6RP ftZo&Q ڐ" z#' t6Uj >4t{LNe+ @0˹̢zeyT>d ߿[#勺VdYQR/x9\;鸻yo˜jh,`|.vRO_0n>AV'Xz=ȷQl!-Cs fl*prooõ_~嵁4LU^{BH{ D.KXªfЕlr;hgDx`RW=rK!#O/yü6@ NM'er ǩ,xP$n"D-7v8 >%7KH1Qjn#ex yz)D8 wƳuA otu2׽ɵv.t #qFuU^}`).~w^xR³NՈ f]=9" .nT!Mѫ lJ$7wZ'lUz61| d"4ȽosBAF936YdLXրxd)7LmlotCKę5J]W{_Ĝ؞g?:i/vΕ > fG O/ $>~@D@ G!՞*4[jrφLDt9Y(W䕓*.:Av^^OldXGwW.P1􀲂;OIve nOێq<-{__\=TUsJCFqo;kg>N8ѫIVhwo2@AM40S@Bva5gE[|=:3(0`}q3_x}:rrV($G*YqT- SC1ޏ/a8| ;qWrH'r ]qbP{lZ@*BkDM7f掠FZ l}_@~6k c+B NDod aYjuyD)(+ Bc[V\ $% mҔ #OAB3>s\,᪕UCǬKWi:2кwI{^%,x AWh1Lγ+8WE+tfɢϔ,,hfssvݣ7paYd2"WIlŜ{v 7ZYz̿ɣ3>Tz=w[65T'KIǟ{kЧ tUa8R=KWbr|1##avGFS񬶯DzV^S**PCaôJу.B"*36sȓV׌w^x6=#SRJ3`|mXTϝp"**/DJJ Szef8SŊKB$/$Uz光bmAz;όxLzJTHt6 9h"t%ꇐ B@1(FgݧBp^O/]bs 8Ieq@3PלLN%:[Y;KS7 p;e#cVٺ=!?v`pr_3z,<L I49^"7HqբT Jv{xw(`+<7ڊE(*AU޹p4U\}_="CxDq]6oy*lW, &'7u[C{;Ś^ym3\]:(BcHу&+ +V] }h3 !7j;5v=;’%ZoS.iI}N殎q|uvt䬀3ϽxmOG}^X JLiĹ職i1?b|2%}ொFMOp&w|&_g?Ӻ|Eٯ8ܛ'g4,lz_e:ō~fgLH2CT3=^CapqB-=R'ـ1Ȁ l~ áC&K0a<}/c02~G:XD( wh FJvAl\ Czz~oN=EY=&̳_I69ҒrYftd3/9լlQ/pՅ 0G.(ʅ#Ԉsw6 Q hDO_pbogIH'0]!&ɘ#X=مF:`kf$9,/>Jgx076ހ+RRzFJXp`mXoRh|zhX<zc]é{ZsIYiyI /=sf0yޱkÈ@`/ZE!i.GzƘ^c(}Ɖ=DPX&z(A%~ Bz@%,е"6oHxy !} WWR1 Z0ʨ=jØߜ[P9}әbűYv)>ǽħ_'.?|2ƵBPy $>ɀցKb(w!b^)E^Xy;kzE.5 +<n7bҞQ/PYJ)'~n]eЦ;^%e88XSȝiZ:O[CAOV U>!ls1 {+X7uemw?)fv$$twG2ͻы+JV,Nr t+KzX}?˛ ITj3j]<" g܈n*']ۋ\L`O1U$@Z(Z!%7aBew24rO]};܄R⽆!#c6x^$q BFY=r}!Ɩ"#V: se4qfTBĴ3t![zx~lU")0,n, v7/%)D؃tYsGԩg~[/sKo[Լ%+O1UeJ쳍7_4]FӔ> '*yH TA׋knlސvJVۼnZRu~c*/sTn+ݵn}TcO*6 }bv/^1 a*KXAdGSi_4*7)^b19LSq"{bH铊NI;tʅT 2ۛeh ԩgŷ&ox\=;er0VT~{K u)Y/ ya81c! 1Y+u(b^#4.:Eو_06н-!bvmѲ]flC]L7taUfpk9"jdׄN4 R)[7[&"y}f .i.kfšX1],I#Lr CΜ3hcֳ|;q߾+{h%Іc{%8Aсr40f\b89)ǰBHBg!#BqSkvT^g#]Ɯs 1}vT|l9zeT?%fl~~xSzzf͸ZC*7|tl 6s{'Cqs%RaI.yH`̧!7=4U)RǍ.xzcoCc_uEbQe {*UMEWqsdG)ZF^Nfn'a0C ?5`p,ypwߦBQ ũ%РΑ&fm*Bސ17‘H%BYTZM2{h5k~Jpn 9Gla)+?'~gKJR'}m#BbMy=)w촿7w+6( +%<&e8Cv&!`'?t? '<涨4Yl_Ql|l{jK;GO}1kK_p.^ZFT _n4GAE\rsKA7`QH8ǰ!@|`<|K%OFEAs&8aKhRpߔk}sLOOk j"on_*9r+k %#3HT?F*vAq Xh~@b$QvK7~#d $Q-MBqn7W>& =$O4} ̲*W6+)6L}x76lLoܞ|ޒeb@HC [3ߵ=v韼SDD$CуfK{V/.~հ;4]{Hчr 3£{7 : l{,L~&-|ŁB{S< #`kBw)BӘ΋K*&f Qsϫ|;~iX(.pu5h9g=S,@F"g:`hӱJS,Rz3'#7]ӓ Oʹ R$|'F Wd).l=p$^΄r]w݈NF+I&$ <B* 8DL6nUH0Dba01wآmr即_k\(o4o_8s5$*-_☴dW,!Gh.Oi(4n]3`:~hPut"Upfclf'qY~ ]&t +vB)C؊H۫:| k!7;V5nPL-D>'T;rkP)آug/W8q+jxP]H>&aJ%x%!Y<3 U>$1 xz>@(O0DjݪA~\@'bnFm:1+!`}Xoom:^\#FF^߇ tLђS#D5kVŕlWmmK܋`k`k+PFB?] &zm[Vx$<,ӆͼLX3Y-}P|zs VAHWLBk5\cջS+΋A,,|erf4@Kh)prT8Ůlݾ_]oZeߚwEtaVr Z#-evKYh>>Tݷ[?aԫg%C O 2K{Ѷ^ u5mhdk)Cˁ! W ?#k<ֈY$XԘ6]^A,`0:TI>>&# cׇHf} Y3/ p!-q;8%t ~^!$;026^I)\:]lcwk.BvhzKA Mr3fЀE,`Cf:{,@ `ph'Է`}c-e0@4 e7m#L=¤I0!ZnHWIؕ e[hъ"7>8pr;H-g~냍3LJ+gշfΜtՄmiJyebCekJ8Y=nF)xE$>;Xv$>W/Mhorܙ9M'."m^04 [(@0 !YVjvD͊Bу. !ݙuq;/{Xɜw!'J>`=!8H7;y#稱Jۊ¦f4$STnP4j?ǐ>7gxq7Vp|OԔ غA:f^3/)IR+z]$컪ű'&]nS˝!Q>[4P lO,P`rzNƞ!ܜMN1#BVU1O;`~^HwkghN`AHǡԅ-ǂzk7wrEH}Nħ[ȔB <\yJ qvFvRʕP39v˹4L4Wtx[??ֿ'7p$\÷H'w}WуI8&~IxCbAP-^b*.Tn!XSw]gn@G^ :@8 'eC%7a@t Dq 2}c7"UAhïuZ8CsĈ }4d|cClKi\x4ۛwY[EH!Ġ صCbD=GW/n7I'KN|R4ŽE*y4<"M-琾Ms)KOmct8| laeуVMxwx{17<̃m|1kVq½h_EH/#lͅߣkD9>vrƟ+qv~A[2rLyccߗv2 +C7a/QJhPGР]7{wƗO8{(FXC_h* ! CL_!/Aӕ慐rIܜ 1/ e-=KX QWoV;㳴-7Zg l}_PWb~+2@{,nóӥ^ik >MCv(tvPqǠ:$~od<@?:izn*uS="w>+zD wp|'|'>ĸ~z(q%1쳐G6S1B3>4FLJ:•v!^z6l>jίXSz5E 73d+QZs/0C! ǍDq[K =~}Rq?i jMW8֣ky#ܺKGN͈ATߙ"CUA-F2cO.`.ᣝ|oآבGfCgXV;`Oa%@ѳBWX=zI:|?ݝC{hҷrsg~  r^3wCl-.*oׯÀdwmD  vVPqZ*RܜL;bC'\ 1opUbWAAA!k,@x:Uh;<ܭ［4,$RAA*BLWk`䉟;7х _VhAAax]ʌx& Q ;v'Vm'AA!6, LXV*e٧6\ NԅɵVzR3N<`b< f*B H郫HX$]1mA~  pBjHc?-Dj+i_I) 'rI|Z>< %  '! Z'Wcyj}|[@#o!;q]xA AAq"D9;pU %DpålFso9:y#z6?=/HC"cT1*R)AAnsbDH_K54d0ib=&hV:Sm.q,u6^AA'Bu iҷ"$R@E4FX@2e5vVzߑAAA8"$"hi1~"jmr;+JB{w$Gm5:V   XNa8W apache-datafusion-python-54.0.0/docs/source/_static/images/original2x.png000066400000000000000000002017251521727353300264210ustar00rootroot00000000000000PNG  IHDRBPb pHYs%%IR$sRGBgAMA ajIDATx\Wz Pͬ?`Y(ND nyf"3bqlٲ^$6x?d-!^R݀)bzrvI"- -}D$ʨUl?uydB U[$ )̲ɺH6VIJ(5.OBdY ] ]UU= ۇ >9ʯ6⬳6 2X/XDkbՔZG; bX{tyH Ȅk9*K6@@Le-V,RE{Ό|'،#0rhH3$@,Dxef)]' BȄr"j AܬJA+FYƒ^R-\C~ aBlB}䂄; SCp]*T:?dx}1rlwyWc:Qg+`2`U<}W8KoW".E#p*/;&!cuE+t Aܲl`*veU'| zEb;;8a[Z+mG]^Qu"no\U a ΔV[SKH ) KZxJΉ^.(WHJ!źziU&Qj-}/wߑqGהJ$zQ@ˬɔ/p$3_xkLB-$Jz@9T F2N`:8yyYpTgnҙIS:I8nav{sEu_b/`b2fȚ2p:d C {,*27w++7g~EjW޵Z^c,!7&!c\^V$0٫a;G,7, "ȍ5nbg5+XX^3]"fiEm~RE!c§s&jHIx &ܼ>ޝ15 U"\kEAwoz #3чMh{!!d ' sdgbÄέ;^YݘiG* A$"ZVZ$26X|iIu-BFi&7t?1Bg0q{z>h#sI^ztMAD7+^ ,eiGG:sY =E^Advt5Y1WrB ymxbBF/b.r?J3O,*Ďr>Hq 3ͷSsl:˦>˗[rg~)mBȈp+kcAVA.\iNNjRayC$_xhi4^6a0@}9;u]>(\Pdi$,$dw0/FQ&_Dt H4;Zxëe}Cp|DLb^zk)bilRvAd4B*R,BH\:j\Z\ls[+xŋXrY| !WW%8=("/kRE+f/7q2GF!o^R}iByׂuɒBLhm)D$$B %ސդNF!dg1ETӶIT\Ȭ% N} czT7|R^m2{?`R1d1SM.HHG/ϳ]!Z@ۚ?%y׈f[3߀$^OhMlv,BH9؅uK?vg/EｳE+-ꭷU -Dw`dR!}ՆZT[:_Ϻ$PR\tewNkx)ůL߅Å= 020J`rA2K, *ồE!NJO8;I5C֞_}kTtTU{Lqv}*o)h=]C,-nKY}ZV! -d4\.[Q|7!*?t'ͪ6t8npMcv;3MYM@2KFE&MV`FVfgM%2xw @2d%yrw,-b@db-@~_#pizCteYN!d[|yN  s]mGv&{ ;.oET[!pR-_s8AgiMPY 0l= b D5Xb/~|}vdXd@E2$>+&gPS&sxD V~ٮ{o+ӿ.-s޻ENሖB0ZX̬?Qbq!_'j=53$0DټB TN2B@;z u嬮|Pu$Ybn+7n3KibY|B*F`YkHv޵pZD3|2^ 4UP) 6=|#7ݰmɤfi`=p+ t͞=1m:DηT) -q8]+G]YcH"j7"0E΍dIK-=v!YD2 :(o)-'0Td5~Ob^L6+:0}nTͺ(\eMjZȉzyAvmG[3; iJi$[⟾Vޝq`z'4|Zz{p#d@[|Y8δ [,'y({ !V`\#0tx)VrM Z>%d% 0Bާ!DomS-й^63~$|wN,0 B;=|ƹ /XxE494k͊<;O>66ߤ!u='2B./JyBA$sƲ ~-w)Q)J04oJ*̾"-#bm}Y 0xs@MO?Ez>>²D.6OH90z߾i /8H@? {uH\ Q!`Q7D^5lpvBٜucE(pu骵æ(CV7q71{N&mefȁߟs/tQ@@5<<9!Rt|d  D+Z"P)Lpߠۧ0Y1 P $NQ}j4`e֦ͰBH Jm,/[vd+!VF7HUVO)l _0#ёg\@2 h8MX#)xlHPICv'Q FtdPEӪ(! @#_n_4:!by!KnMuf5m]!Gh]!`H 4LtTAQ#mz*(I@'YH x_yGfUVnQAKŅL88.JcsCh0$t |‹'(?m:,BHW  +%X 22|!MIj/003`$p3c:u*$&z&]o3--Tyޙ?Yf#E@r$Mx< !ղS7!$qX[l1 )y%Wx[IBT]n0wS߽ZK֝[oc H`wN] B,g FBs[tҙUt"\a8e~'d!jvTE}fc("SkeYw]-95\ok)ko@ȉu v4 b7&9[d޳Fcb_S,*҅gEguaY#NJqQ=MH-u6jQ0]u%D\^ja ]` `;IL[Q943Zoݫ_C59]L$.N9H{D̮h|^^c{cG1Z.oNkSJ0י׏1.W W[ԅ q,q)Mv;Fj}u(RK?U qqdƥ!DQ 93YמD@6c4V*<".®%yKo/]@8P ߃D%WM'%< laiH#歶7س5s-M L,zV4qS5zpZ. is@7n:OMY@)/d|{}Ri}b|{r+C! DW{ [(ؠ#_Tݯ.] Н EHPש^ + J!S)/|ʜ^7҂(jw0?˵vl@hQŽ#FQuRqeA_P|FxK8RѺɤ8 J@$0(Rוc(V ~A/$5W D i]YPyg ]?q)jrBI{¯=T W-.g~C+|ryFG~/Ĝmڜꅦ2k(Im[E (""&3Q<;U3Wu !|6SԚz69'@/ڲ"ۧ'kԜ&đ` ' Rh*uR􂖾}}>ֻC>Qm&F3< eo^t&V#\wmĒlvwHp\6bu5E`xlh!FDM4֍iL򀺲s=vze\^P/B渰Nbzw{G@7TFq)6/k?HB=PX&RNQh[D^ Br[mS֋t!#磨5U;՚1ޯ|~93쉂XdgSg5E&ǁ %ׅJ|gIPvY$YH6ޗ^1ة~ͩ8iَӔ0ێvI5e[̿AR$xmW"Bؖ"m oմ1"&k355 !%;J/1KO~̝ILQCsbUfV ‰@;MCSB ʞE|S_J8=v/[MB`N/蝴ԟwKDž`ImCBLǽKD6܃CrBgwfH2ъSllw~R/tK M1>29 1[ kJ nK-HXE]6E9]ǮsQvޱKo >G;T)zV3& ]]g &6P!~XaLі_}=_>]Ju~f$ ԇi'H:{'~9-\K E:ګ.L5Ts9!/4e]=T7N_KaQ2Z$]n;[Ki馀3vJN5;QϡXh 8CBE{B>h]ΣΡgTecCjt0RxMIClWeO]+L>{Δ-W hLiQDQEd@d9g 7ff !dA+ J|eK?5oRűvWV\Q$ Bm[9[(\1g]-}VzK-ޟޏnFdWe5i`g;}b{̝(т~J3b\ao1ǾӨh>9i+7eԵ޺]pa&!!&*dkԢ1u6%]~+`{̘IiLt'BJ0P! ?yXwLEscBLpJz:%ЛI6fևE|=b ;@m yM{{r{4Emj}sYdQxdI]ԅPY\W}7kkL";$nO6l;2y)mҧS-c -(,#R?e'ڠ!/^da.pbױnWt"eoOu=z!,LSZPvOg.4 'ā!gh P]ܴBh:R,$`D" Hu#_u!*2MKmR}v!5>bĽ_~̾{V #KEw]zPoͬuY[Ia{.$"B݀qߛ6+yM/jrEެ91hQĈ *^VM0p7>GjU['޶AF.h(BH+͟EvKk\cP"m?W6E #҂Cjj8l&>p;FBJ|#v}_f=r;vC~,;gSU 6]Qt/L,%>@OU]Nwm"Jg<$suCEk?Bb}Nvs饅G alJݻ?W5k¦#A\'Nн2V_&jz+mR%[O^sC!܎m4)!֯Gb|=tG,mB&AcuC(D$ r39 ^o2Y+v>eGZjˈ;UWEvQX}aoQ6fS[Uxg[{4M>HĶRT.u\ӂȊ6@|#;.y}9'ҙ%DHw'[wtj'M$?9u2۲8nֵ!ln:R'X۳EqB݅^;`U)jd!꿥f67v*Lh~?wY@맅;bjzy,@)˚F2ai#7Vc6Ya]N`h0iqP]M80^/2Npl!|)vf=m8`aV2쟽81Z£H-_(ljEiM2}oޞ ~?\] TcWfכֿjQ`济ηFqZ~[ݙ+[XӼ 1d0T<7K\ht]M.4 smG#Ӂ(Fz 5{$0"Ko+2e /~f7_W;g9 QZMR}Ho~462:]0e#sZd9O7qarH]fZ*%[<($[1dbHx̲k\z,4}"0}/&8\ZSv/WhKiOwB'e%dlS.A׻FUpgkkRυ/BavHPDd;S9wryӇU"tdMĶtab׹T˯nWi0zˤ.As=EVR㇧ !#aݔz+0^ =n>xo BnmQtnM}fH1E=rg_n:AJA mEg|K[`%Տ]w/̔#de?vڔ/EehP~mOɺ9vwe}X 1G5->mQmeCwTtk&k1QCۤ1Lkeg9 Bq ΐMCXXxHc Z̤|хmc ٤%I7s_x>4u5f\hlci0|41o5ؒ>wkq͔8AM|hCJ#@vwOwk:? -cS֟xu+HU+Fm;WSEO5^kG٥t{ fBcCn%f,g9A4P$Cy }aszLu2m:1 {a["u"W5߇#ZEjrwzNGńw)ȇKE,ݯ?k))Es܅ <~ls@Zp).z>7nVwk9[ǣRm%E:Km g.TKrZ]@RgjPhoeio4[jqyS;r@sfJo`cW^YDV\ub. *BdQDPb'˧VBب)-fv;[/u'vs1ʩb)47 NL}w-IXEu+Bh֓ B7u+΋i}SJυm]{}:ArDXv⤸I*W϶KB 3_I"v0./Њ9Uݘ9=[\Y|~Oǵb}&`uFѭȾgnoJ{Pu1bȃ[Iή.ud' ua'|OۧN8ؽL fRזU: O ‚to{_.CD8NF^bz.FMWB' G"s>({`E cS߽<_[ (fm?YYIbhDW˶HsoODԭΞZ?)0nY" ( S(R_/aA=֕T$[`1J1.j1ld,s>G^o"/xhkLS\Z}yOD7&N},n!=iK(|1lOkY{Ză*Qf?ߞ?jI "o76:*S˥g+XGVkxۇS+3ҖZf.7̞EʯŇ$I)wks3L\h:#30(] sa܅7M' =X!Ҳ.XYVnoo[=y36lq#/ܵKHѥP?V2abWNgdxVN,HTdi9&cV:W9 cU"^,+{u̚`tffC*Z1̵tC+*Y%8HD*РHFbD==o~onҀPl:)я]H!䮱jf>8]%:Pkh̨ډ=.BK.s(bsPAb:@8Me SR|S~{K3ߨvD7JmZ N]JFDT;jZ cjI,/RwOsybybU 7bBԓ-<‹}c\ן1-6" H!$b-,٘J"H`i=?BcH_ rP!O,ϏY,[sV,+O?ohQ4Ng R;;{gY׮T^yÝfY=nYKhu-8~QNrY2Jc.hBX5htyQci!6ȽOxTOP!t_W}u?ﻢen]ATRq$_/ (G/sbm?'YLOM) t[ Y?x9jH'6GdzkKގn7dJ*,.5=2‡q%k )]O]2bcBJw2{FUC YGͪD<ju߼#\jSzgƐƥ\_ ]$PD-HK}ڿަMXlF;(jjmŏ0>Ŏ~W Lq)TckצT5D6]fj%,4t{Vbއ">?, -6+,.$+[,SwnifA &S_1|^-y4{8jg /\Jq$fCO}0574W:!L7WRVWcD𶴫W\>!FQc" /Y!ٲCDw3HW?2KLemh)u?y6 O"3c:De.tN Rk;wCUxPDooZ>!´#/S4;?K~ۜI'Z[[۽Xxf:bIt= ^3x1AdI19tDS$l#r1^5GOI) V7PM1u$%^~>{/7YOӊ2tjevu4vI䦐JlXZbHSo: ]-Zӳz2GulLk]GJع907X3Xc{9 ;IZI p3:neɲzLkc *s/^)Wz=.eMJߏk`I]xQKI`F2C$![voxs6+.g!̭яdW6IM uYZSZIABlWoϳȼ~+>\37w72d:ux˴W?>nmY>cEf+͹ivjk ^r3F=4?X58ݣ4>7LCu ""2(u]?څlups~M"uٙv^91~;bħ]Qb7-ILR`BG߱גwl'$ JGKŔK Ű}]%X_,5=Vtnt8NN_j=9ZWwjr^JΙiʲ7‚8m;Q]q90O7dB#$̋/6 DkA"<[`˜_o.BgޏWWX ~ޞxf~;]zoFGedsDm 'M^M13^)M.SGn&<')wZҎz\|h:` /_h@Qu3Q$ Cf!/ sG9}#$y7CqMp.+D.W66GCY+ˆ֊:B~/7?P49r`ΊOy ayQ"{;_^m9d oBcRfƋR*I# X.WXGM/Bh:E][)[h؈z&sC}OF 9y\#B2[k.}irLi+QtYȁn*]S?:j>Ƚ_~y_fMyZd|کO}mcＥť.KPrsW "3_jjv(3s-M@4*9[!z($:ᬫӬd0yzw/`l ;bBE֏*zl7eEz"n(:B/ lL>dUfe m-NBVaYH_R@v{"+b⢍M[^5xFצ]xu&4p.K?j++R|FmGK$!cxtn/>'FOJL*I4hnlgHfAբ!cyw&r!f~ЖJ\N7XxQ]tS IJ,Id<ŴzqzzCg.d-EO;c݇~gk+o!z{nV+7./w7?5!B2b81P1wK/2) 3uW^8+g} xsEpkՋج.W5!4,Qxwt\>~']]C0e"R/21OX4Y,53cmd9"9X#]HͲvk=c_0E7I8VdADq!o__xc#NZ, 3[16~GNM])ٷ>zw9WdGLjgaESЧ_fiLxyN1^Q5m'*[Viɍg;f0N@a)tTky7rGDi2 ~(Y3n@PɁ yPzDEsNgG>!YwdgC'RwZaW>ܗF/$γ A5Qۻsy<4 B[{||89 (r@Dw:w, [[(vʸH29-9IHhcL'Ѧ^/&{[-='ߟuc>yn˒$C~\iEz~E+; {>rF8%\}&fC}g&gZ g/&(3\쏼X]>ovYSb3f-nOE;_#06L{k܎gSxۥ)il~#}?Ʉ7k_75Hɖ;1B '}uo!]>Y aErG| k RB&ztyxg=L*;d7oK[-d9oHq/jҕkl%4X18dFqvaT)ES2eż+Of/[r_#?TN L`s[DtLg]w#ZŮb;wNVY{ȟ7݊( 3=g2'A4>NA]!骍"Ebq%HW9yQ'BHmxO<@& c\SbR'݇ZgǼ+$bz֋ bkُ'*oBH B2a$@eᱝ_\v/;`8n# g;&|Im>oCN">;DTTVu}=,wLEGtt_C/:CAD4mXzg5g gڧ;%0!Qtv6 ☧ԝ?|q ]`:s>jmcrjN ɮ7h jZ ڟt}*0+D)blh8SLcjsDaFH(HXͮu]Ԑe7uyr-Ë{m=*gcLcLzSz]ybm#8vն lz]Eg/%twRLcFE2au^X"~2zfe%ۻ-/b8y-!>O?L\4H>:Sd%X2@2(R'"oGD4kFbHʋz.ϪGe`|F)1٨H([ovGOdKo5yc%NtY+7UVZRbS{S"kaW -'6J 5M4;bBlC̸r-$swNcЩz1\|6RP]_6c#-~0]2~!4cG\^mz1d\haD|5Ȓ`_і"cS=WA;F7sI8ˬT8*y7$|7Cڧ qHUPr <Ǔ !˜ qN7HGN'KI$ty/6`}Fa%-}=%wv0Y1!R[䚵x<PJXHPmRxF{ǗS;<6 PAnOU]OYN鿗EfבIJzyƃTn}WZk QwN_K!<BfnKVz(]霐O~m5 r_SCE$gQ'vh]Q 5wVl&1(&DS)LJg` gDxʊysv . H?6bIaouo/wtSl.np,Va";3T }09Ŋ<-wvVF =fV|ʼ.7?6 @u BT.;d u-ṕY_Syɜ m¡;fASBțy׃O)@rA j.)Ͻ|S8AAhXtHa]^тإS3ԃ4>PS$)!y2ebHD׳Rpm3}ɔuFgd666-SE?/ѵF>`҂?orN\{߇8[<;7A@ ٘S«gۯ+V,D!ZH!F!/(JN)q:s #;JB ѥr&PB=,)D%wNC]kgd\/o3w n\$gm.A̾][E0f˦Ϲ2_LfQ$w&sTv4i2 zYAv+^:;m 7IRĦr (]!bDHhSH6ٍ@s]g/g{믅s-Dw(ݠa֛YJx%+b|p#JCFYP>:Dռ$ ΙnK! 4 :jy6\VWȎAx)XaH??xQ"Os3ً<>s˞"9j}D~q=-/yu-׭?qyhb IO)1畷0 B-bzbt4bN_k^k4`@* 0x_{q26=V/'HnS`]~A@V> KIO/kZG9ik,x}[i1cᮐC/VAa ny>NAۅ͍Pݟ\;lsUgbO@tP W3d֖m Baf)]cun!mQmnZ NGLW% qcֿ~̜iGB"UJ&RJL})`E2 u+Z6ɂsK@k{m~>&=ROkԥ t< R~wRROB!CLfHfć/tǹZeV>=ruK @+uutCp_aD.>dd&QVßV4D=+:!~=eg .ZM.b2CfX+Wzw\yQ1(Rh9ͨS !joA:r(&:}W+ZDMmG.=Pt;zr `jP#7bpb:""ۅg'bA8ɳw$+l4|h[ !#X!d6EX 0l}Bv.Q{*s;@qX"WE>|=.Գ>wgi}O&D;C {?FU^,8#AY[ 8:SO\.N..6#+j#VNC$jM}rA0o='Knw y6+0u+x)bABĈ!;ɒ޷o?z:"!jX:9~ ]%#kO׵f~!M cNg?jsBE󳺖{gu~:6l&G[h2];sݭz}ݐ-]V"뱳EfߝnRd0O0Oo^<ԜDxpuE?Q4q[z?NQD7kdE~U/ )'eA\/QҿnGs&ELiE!.Სm6Z*͓vyW!"7 s=zfgY[.#5K?CK)pR"J#v`nֺpAyۜ[LN-vYMf9M|;ӿv%ͤ#R :e)Q.}'h q'&(b)vemgئs$ea;2E< "uU3NbʓCgA?]X=/7`@5J1聽1%JMqEֆv.MFie^_}{!Al;GQ؉AZٍsb_L|(O !Fxpj;CA)(:ifލf5+.TW'~?إf1jq42C }lt@W4Fvcsj[6~<҂]b?n>d0V=aNxn.{gޠ#,?l Ҁr'f6:{yhďRq p'ENj4Fpu8nzḇ~tQ$Ѳr"$/h0qޛ⚵;?}?9h1B|W.g&@ƅ ŋ}V3O1U13 *:YBpY[ry"#|?gEw2HƙoHǺƮڲ_J[ HW̳> bCÅ=(b) e@/$4F{=⃡>（S&.ۤ0l䨉)&4`) A[4"hznba,*DB$M't\vD*j0:.{|1̆XQD_/-T{|LCO|?G¶W!všEV&2WDxnП}Sj+i, lr"}w}:UdVޛo1"fB@]!;D|CHvh݅2aD;cH5?xuEWh.u)Nr6k"S X gЊ._;GR"uKF9tY!*[b.p/u[5חnwӻ Wn+?hRr% 8_mie`i0].٘Rں~&^?l޷g~5Vm#H l(e6YJ9%vu LӐqb:0jC?%@5jN%POpCW&9@|AB*!HL0btIb-p,/ഢ6SggF2;mO_zbx$G[ekz)Bƀp dIO|ӈ ob-tDљ&|m~H H[K)$aFyz)Ң+:ac[% V9u$dH-ZmA@߱uA;r@PT ti]Y6uI#1@Aˤt9|BS dZ*a *znÑ҅.Ezpܙ+4Ƽ_C&YJ-2?c"h} @/UL(/ J dt]KY@]gqS%=("[\f}?8_8U)ScÌG& )#gh#|hTuX`j8}R7HBA風l{}#?p$+.je_eI΁|᧯C3 iSlR#ue.;H|N|>*/-`8DNBa`BiP^#2+,DHwi`To7pb$F£ĎCAZwd!dvck[(Z,̃|GqE^@UƏhPU.H`GHwHaգ_RG# ƙ6\rs'o#EPzq;K A@%(X'(zVMx@X38\>Nu8;8l@Wh9~#|&p,nwdeK\ѱ(97D@brw !]2(*z~>%ѵ$ƁWp@qZHQ2p!mQ30ڇ(tc[ٛeSXa} \&'!\#1m`|)MBBF# n] :~ Q?Uvğ}nVKBf,8=_~H)Jat4=J"6Aޘ{Te_H`E RGj<#=q&:` [ZČ0r73񇩪 0"qM 1> 'A*O7]*"j|3/fi pXٕ"ڙ8XӠ3 Oz m"vd+ؙT)ZR%\DD@:U3ڒLfSBb$ U ]A D1"kD9EB zGF െn+xEx5?v6삝)?=ɬ~,#r-n`0˪cjErG`%aWbk06H!#`lqEʃtc@+TeZ T bEBbD2Y^D]At;CkT퍻 ,v*w0cA8 IL7P|Z*Db1Eh_븼ޭ~BZdl.9yFQU :4u'?R"q vκ^`^;'앏羏0 1oM3;_Ov fˋ(" HJ!\Pa#1ncĐ2UF 5C7hH`7E r4L7Jdrt`5h#5!ZnnC_c+YVpaz;BYg^K)j'蟢}.9gAFhp/gdI2n#W= q+wf0@ ɈPN CX8$8920a5 Xd 3_}B>3.&K;C܌jiq@WdBfkN/#& \p!f;P Fl?ݱ/>Mcy.-*D$10@ N/ 1rkcFB# e t tT^xqU1H`1bt,&BS"M{e8;Yt|pZE[N<<.@ǥbv{^-uPi&|!`:B>oSW}eC((Dl Á]..JYB 3bӨ:41` suFQD(z^ֿ13~D XdVߓ&mzx5u$uvp0G<?J3ǩ4O[gV,dS?izrEʇ疗{"?,dqYTYą?P݂L qg'HqwzAdqW5`bȅ>n} 0IB#dVB8>8iDq"NJPqL&Z?Ǖΐ[P LIͣMKW"Mƞ;rTuHAU]nH.Sa57ԭMs{m$BLo~S5< !e텠#m;+CT8mQ| Ē?OFK%:?B0sְx< `}\'{._sS \"e1P)n+%ޣ`ٝULAFsT fwza 缮nQۀ*UL[Ϸ~&֌ccƚGX7ۦgED*wK˺ qAt3} ":JFG!Eqe c!SÃ;_x́~2u@Ք11"MɦKS1 SgҦђ~xnEVL9z|5 /W1E_*t) -٢즲7_Y=ogA*ʲ@=;3q ' u=, !B%EX*N3y)\W!.a ONtUwA&bĐ,|?C}039R PYÙ3#Ac7R%8%KЧ%뺪\{{,=Rb~DL.ч)F#-fX@f0-KUNCXَ-"w_s=tTcEֿnTEĉ ( =QgI!j5H A!+Ү/?luŐ_tE!Xyf.CiV,\lG -=E sl$|nX`@ 84nM6|؏Wh;*RBB?:h_O]eh'ǁo% F28a, u W%fV$4^T"YN8wEs"ǝq{Qq8`,pvMn 穚Oٮ>V )&ʬKD&_h@N =S]SMp:!/'O_tϱ*wKsअNK)23hQïq!o?.AUBEPIĝ>avꅎ-+_qWbm34&=[ ؖ{^ĕZdzOb3 F Әḉ; a&ӈ'<{C:,k{ eUJ7^@xoc7o9L.Twq&)YRl/A9Г/ws-k%3_HS}RIi`0Y7S| A$/86䌣?LA C`"1`@zy\VYZܩbE,,ƇuԲ@ v b9[rcs˪">.&{C3#bRT-ϋ`P0iOXS^$%wLؤȔwm!CJ0Nv$1_6GEYcEʄ,RO_x1\8T 1YÀe Y8TpDk3dSQ1EށE֘гbA'Ü q6Muzo9 EOi:.L1% ^bv!k-nj׏ Jwm%ust(;½Ws. y˕¯z_U1軺oyZDfDffPuqϳN* xkdBT3b^jQGWH"k<#F7N``gG J^ו}a=AFC1CAAqF/*x/6zP.R3O N Mʛ`on[î(=^fdm>5_ݸW"s}a ێ\ a¸Gy&&IhAh:2O)NuZ Y0f5A7ӓK%Q!>Z7dž{4<XGHI@ϳ!lʗK qt Z:&̴oeevvD LJ?>8[fW[4~{kt fR->Ț >C!Ҡ頲IXBh:pvSп-Il "D,FHnrlz|%׊!p0<<ϡԙ/f^[b3C-.ACq۝ЕwAhfT.^z9K(rō3cdĔmrc)e) 8Q|w>[/ܣ",WiB!0+4! 40}[ccMU^R GYMB?\C*c_Tt`0Ha0~Z3#nbn{Gc䷋b2XfL(NqTحP ?Rv#RzLX;yJߣ"* ŕT5S+4;{^e5]!_Ld!MC (֢!tAppZdlq՛8 Xd.] !5-ۙSA2izlG1wE9-NlvZ=³XVnE rJ^ʺ@a`tnCA4ΊNjcҎ/.o*E-Q";ɻB:[C"'|)Z?!76$ի4b(V 06dpx? ϛIGwpF >zQEֈҕr?ag#RLbH5j&G޸L9l+-E=P QV] yuj.V* 1RgͩK nyApI{JQ$s`iaS[\ʅ#RKBx'Rҏq+􍊉!(֚tI8lˆ ߦ!!kŒC} Kh{ l> ]Xk5!O7*be`=zD`5t O[u~CV:g+g#]!%Lf ޘK CG1GDl0ZAHɦ} '* _⽶B9 eggOGt:DfȈeP$4`bOa͠J4hp>I 36u5I#VlE_=błUYɥDW e<E&Ybc1> ?^Xpi!>xϞS\/$[5Uɾ2uEċM~pK,f!0sGkkk[^* II?+dšAU* mڝ:8<8\%x?KCBW3 eE9bP?1(/DSHi<޻J;|Frļ>n2|F3es}1]ȮlP '"9gd4;$Dm%~nwpH!}15ߖvEU 3ٽS &tH%;}2GD 8 šAF.`lZEoѐЏnQisw20".vAw*0)/=EhX!nrɄ3lOj]9I!&Oz:D8?ȋ m넢CuE95/vVXjM}-cŹ"E؝Cu'>V L1l.D*Xxx"mSo$;IbT`6Nےܢ? LWi<:B)\RMwȹ /~& '35e#x2hsLK &Jh!D$"J4DfnK볭L^q%V&^ <\D|eًR0#fYaΗ5 ϻڊ}RՅrBE0EzkxGh:/6T P2;[Bl~CKUklt0h8ԟ tvCMt0U&,& G~eZ֐3+Q> MP!nQ&d I"e1$/v4:\ ("6Bt(6Zjw%,/ 701Ôd68!v N%Zo?2.<@ٯ0h=!5 UG5M *c b,!2pBCdFZqY[ͥl̬rzy̲/!e͂sS$9yɭU4 @:2wp١xr*Ԧ<&ᨳ+iUmvX.fQͅMq9!C &NwÚ `,O 8!C&CPp3ݫ.b!D_c9:Dt"'HJ<ʋE*v0!v;Ő?2b2 f_.iPڰ?hӼd]uu ڎ"QttHI [y%*'x$[CΊC)Ƃcȳq*֖~04@XxVz5 9.4@8LZoQa{C͚ 0ڰ=4LL%4:U$"*T炻ɥpdɊ5עaĐ ș.^̕ )(Wv˒|d>`D+V-!4om!TQSs(9bk}cALicFF=]-$M:/l3CafPP\6?8{7_$"n x]aMTNCӢk$ttwzw=P}ZpC 4X,sϽ)e;`eVf(ݖ!늑x3ߩ֥vzװ~Djq#;@ 4L?ǒuWd'"t#$@2"/F:NS׺KyPվ dum[\bYO&#AfA L"%@õNJ"Qn!ƎGZѼW*ZWš 0k?qeQlLLA<̮cS"DƄa[4Tgôn {TB.y@IP*1o{]?P1Qs$b> !M q& #WeE_ 7{XL{D\[/QDyaӾ|STl"+c;0nnxix  +=@O04Lm4BD=1R": փ pq̃4I:Bר"bkeZAa{6vg^Ӆ2l|tKq\[><,%?sFF7،=vHO $t bN=/k}z8"۾4&kE!ƙޭ>CꩠfhH@>>fm s]ղCVpbZyÈ"F iѰ00dg%!S.sJwt2omJXL \ķݜ`" DrX4kL)JBN%#_}MxS" :vK0vܘ;E  y6 7bJBHqšAp1 f F?aMf,Qs/s/Dh(8214g5]!D |Ŝ*H賛4b1ӲqA|#& y$rϾҜկ]̽+<A.,v^9z32jLE|OE:g4d& # &ytN?Ii50C`(030Oĉ"1Jk{H h|EՁc&p֓@EVEȅshۅQa eO"K?2wWD5?%u#XR el4X(PVYި] mQn-/d;/qI43s(Zs ;:=7Ü3Y\v#LJG`a qxn?ևNG!40X&ME<B_+-GGx* 5Xd#2SF%0fݻjDZܙP:<ͨ7DO8֊n+ç_`uY%H,դ&Rmsj_9c\ߝ[&Z'~a o?GfX";`f0c!Gh:`1HJAHiXH|"pT'8Y B8rOYH?ڒS4ɭ\dpzJn8q0("sQ7raoG&…uzbH45Imk.N/渳lCXga'/c!v MU$!SQx9gև`|{4]".Kqh>Ĵb-7i ,L.{2[4nR5ׅvfDO>O?6 IeW._ga #;)u)'^KT{uZ'Sb$:B@bG/Ґ03ڠ -+da5 k'[@BDX;j :!`cѤ! KY`wzXH(ӻÜ5tދ2m5 a95.ndVE:b+2׾\lpg9d]n~/}&J_~uM<#D7N>l*̵eVkO7%zKlq! 'V֝ЀCdtQCO5CjҀ91!a'!b3&\ qY`uL̤q-л:ר"kX!MCfYFiE$t)TJ! ޻@pgm&񃩽;ڥAϮCz%QQb`y%f%rE~?,zD4mQ H'䄀2K2 HxN_f0 N5u!c b! r2h8DLCsYC !LќNvrEcnSXZ2C#fs>)o9 ŏmo H}1U-"nz89?y*^`! &767t,p"h*854 0hx4zynl =šh 87q{,@h} !E t蚘ƙ3{MEDc"!薧Z4I Q3ۡ\e"\z°J~)޼,1y1B(/ gs_AUYg!5KgoTaD~)3>18A)''0fa̓1BjРauE<4aV 16YCnR5h:`" S`Y`jx6nhKE4Ő3$$"+%у9UϯkwsqDSw>/c[]q8T Y&HN2qEj[}(zl{C-2 ]m94۠f!d0 gvO4O`aDW1S* ) bmLh:@Ev,*zhXAbt=&3 3.Y&2AH`Uf"m&]b'(Ms=e pk EE;y'A꣎ DW6fATx waH`uޣi[yO7Ӊ=BJh86hp"lodzjaÚLBFÔ :lU *yH޵t"kDw(psg,%H 빂>!/rw[POevwAJ" d\pz|WvQ-P)GByn[KÚ8Nvޖ~ [^a UdAfv{]ʱXdEȶ0>I[<9`SjqY|F;҂_9[珘%vCF$W7z&nD׾w:04[$Xw˭q Ex zB'>w:aS89Đ&&FT%BYKIUA4hS:} Sc9îJ>ؠB,(iFvHŤ_Yfoj)2-ANpkk)PQ!oŲzQxx^U9\ 􋘆G iՎ'a`DJ<um} dx6ust\#"# @}Dv<.NO\cET,LđE|}6Q+neZ@:n+ *r);ȈY?/}xg3> v~~f4}X ,|hB S!Gd^ E2Bǽ % ~2_ 8 Fwe dҙ![ c@-]V&9}|:~81 Vk2H5/(GYw 3_=$f7<Ɲo7]d`}XP8^BǵhO7CϯjbSV i0K`&AjT )GS}?ѾmT3SX:H|@~Ӑ?[4̌#}dtЄӀ@X3:f1,-+⁌M݀Ђ!`pvY.‹z: Ük(a~giфa=ufxNß;ENE;+ސ9[b{Zw*HGFMx'fK\4P!9?BS(`Eb(ޞF2#y~>hRCةm3# >NCa͠ 4`WU9 ifA0BY bf*hџW 9C1c1 M2gTm``ػȟ)PZ4(@(@$X1E2px#'$Q (V2aOKo;DܽW6 b{ b.X0w%oq5R2_6CtM7_,&Û3Y\2@ xMFë4`"5 dPQ -qu|E]N$s}2d@L`(.kṮ]'OԹ@NiR^[\6l*SIז.Bt +/ҩ m_x'?0bm_]ٙz;ke rlGd+a*Bo}Lgv>O$zqO< "nT'c"bhssf 1IZgAƁbX3BcpFL%pm>Cc=oDH:d#;B$Rȸ=*UlNe,bXd<Sd윽~7ٍw̌) u,s5[WP/+eH:I^|]ߕH?>>?ŗRa{sB8{s*oq˷2q`UQ=p0Td5gC!~ V!psŪIy]!5#4< =4`1;.N` u~s%2AR=2ɔ9Ia"䉤fH8  'S[=U@gGV: RlW%tzwuC{(o4dW6'MAۡ{e޳S'[]cJͽgbnܳ҈cf&( ewrU*?žsHpR sτQDz+4xRk&C0]Yz^ޡuq8L:qf>xwމ"fItP3B?9t^2(vkT`uDT ݞbr8 DEpr*WQ@WGqcʋ:e SnT>ӒUUїpKD`]uE6`9 ;teDCg}#V(tE #Q<NMM5v`NJUh '?\dÜc͡x֌tp ]$hGVhx 40pu%YgUdFIHiH蝩vBÄ!xZpBv:An>f{ d#а#QȷhSKW9V5^YJTC+yƇQϣԦW>xguVk?\[vAr\ٝ9Y/ ?!πMpFfl ?ҮЭ<֒Q`IbCZvVæ!I(hH<2Vļ쓶n]yޛ+`̪HtUeeǽU;=;,e& /p~YU)AP߹ͻS@'C?Mwzcu$ph5MZyb uԵ"m0DEĊ]FubWUrn!Lgi\ /QhGJKtՄ uZw;8c|v1 #^ܑUxnh.߼v1iұ?o|y7UCB[ڸcV53jE0Ķw(mJFgiosʃƬkg PᐭeO#\}u yi/tU1MoSx{iu88K.U8uF6\"cwG$uFa!o]U%f!Z=9r<*YdǞvC Ͽ,I&["MO榻jNz{MmpkJRH Q0BnFM&4>11vc֣"2Gw&LL;tb KTke_srs[%Z\[B#ĕ-sZ3,JQRG{AȜoegt&/Bɲ<>-PhiȰ$bt5GYyBujtbÔtuU㬰G)>h{<!͍/ ]axTIԧiPAV6%9"G#f^1_d/uc&um0] #~f ,rE0>v0!҂\Ƙͯϵ~Lڱ<.YU,L+(p8G0;3->GCz2Z&{%j⹤Kdx&\ƃ#z&걑1҄`&BbKrE՛vUrK?ڑdP7Whg;qH;?{je`ve[i>}@<|/fobʳ9\ ^)cD{H;v Woc{ٍ˩Yo^X! 'wcNʧ%YJmu2r=HC-.)eTMFOY3j&O0^&xI?ne TgD [\K}ۍM*DSվX{A2YY_+GY?MT>q|g[!RwVǚM֗XdL= yD8/APҢH(1$U3`Ē$X|b‚EO(xD>ċ3̘:JYswwc<8wy$f3?&v^͊S:keO~ԭO~"ZtwD /uKEyr @3o2|i&/F3 feZz4ЀhvK;8sL?()#H/lkn6i&`'dǀᓗyX(Sꆺ/l"0^"/rKTòY\id5jˈ1b F^^Ysn*'iɄ~}\.G Oŕj/,EEݜ,OݲH(UXg)ke72T +Ұ(e'h4/WH)q u+'T%Q  ri97 ;GVz) S;^M*Ĩ.}-ղux#& O4m {j,]傆F3HUJ<%F]c^?A1{o$g?bgT^n }POO2[i9$vBxfjc ux`}MFuU{: RDL/u}:}&+s)FjQĔɍٍ:FlkuG!w'&y& a~kLUu<Ҧ=2ov̨~BTV}@=Ծ7V) 6䁖34A R/;݌9%>1KU1(zNLVh%th@Sj]PVAM ^T]׊3|23^N[J=u(Q74 f]}&P7ڭь~`Z?`vEnTqƍ }ym5ɘRYUɇ'uض['^E6t<-B%TQ(xF}B8 J.;iaҼB]5OjR)> />X+ 2M}c1cYRE(dy "GP-Jbu$HIKiqn}\{j 7ötN JU PH9#I$Z+/3G1$mR{v"C߆F搵sW{=)i8y%4 LԳ7-#`:M=i:TvSu_` F I?zOwT/^_Ay,}&DjtYߕO)o,9UF'~8`V }:2=Env_ލd$F@A!YWZK!%PZZ*rsnYuv0'HԴNCݳ_\l UuxٶRS(Ĕݵv9-D /KG3GݡG7Y[c;ˡMneu'Hz?e`&#x*ҥ4 ja̚'}_6z"<BJUXJje`بp~e|8k R qD%F /LnKs£I*cH8BohHBn.lL W&WWbVqkHr7"%!#Yܦ>9{8N]__$K5pK*~룗.& q@a T*iT^C ~2:~Mn[ LG~k8(uykaEq3s8i<1Vh50` 8>O*VWgǡ|LՃ1a{0_L OMLյe-6J5xT~ftDhgXjxnD($TJVs1!}pVb+ Y RMžAOǫ)R>%# (e64v&Rt;cRڄEqwTGd,Z-it]Na)ʨn2G&)AR=RGA<؊_L\pSŤǪD!~Z!L {2 A^">yQ# Ԛq <}{)UGsk*> r3".'@JdI}u")=E}0gJK f&?C`zaGS.خZ1f쳔*s\3m7N(2;rW|G*ij o\.}K}Y!Gݿf9ޚ/,ߪ8 79 ^P\ )etfu-1df3,ߋ^ȴ8VV5_zZ2Ə63:Zl~~,Ӏh)]Ey|ZNl.=z* jQ1]Ҝ0-cOD7{5 ֶ?_(a buO@`#AA >N./EL %yD(z$z,xǫ_zhGMd%laXQjUX,pKE9*TjqEq&p6p~Q4[R_ 2; lc/l+H&vYhԁW.UL?u#4Q=/HaFw,HKjG|&ŗ^}˦f :9Fp}DiqH`$iKVle: hϽEZk 5G夢wł옒m$ƥ \uA׿2_b.p*N‚j4rYחe]!a˗2bt.{(*ئK}̾\"JŭF;'upamWI}̱oә^II% 9jK׎V5%A_e]&M?$ԓ ֫%>PN]뾧DA9cHFD'QG1HG=SP8KjeoLl~06߭>sYVw] iYI ]ϡBڳCY!*-B&^^ ZSMQk$\"\!*# 60DdFW\VJlzJEl= =|)hAncJ$Խ4KqC;T395@H2mז(䲍'(YEZx4\~`ܽ[Z}kvĝnd6^PBƏ6D*F_ϨT['Nɨ≓Rm֣ehY 1":ݗ"dp貚}Ԛd[24ޭe,/%B~yQDN\%$2R$̯ot[et2wul^$@L&ϗH,1>& JXqk :C( N mZ#4(78bZ]YL5$gDGF>0{`iZAT;Pk-j>yf߉sэ8Ȣ%$y컢G " 7 fֿR+O.C*%0pQ А -WR+OrJ^SQy(1OPPwTV̯&VJo$*뵡~ysd4LiPI!]S~({+jc!0\l?1)Q?<^l륹EuQP]+9{d~EJ4W&SlyLP<F"(iTg-ѢNk:׮.%># R)S:>$d|9pkJekx-"Փ߇@u+Artf'//oD8ZHl;[?,e4H9tȳ|("n''HT۱کTGi'b?%VsB`DOer|O$;>,ӗJ8kkq<=rL_= \TKwo_YN%S˪OX=K_EW0!M-R3ewhSdDx;Bc.5Ht'ߦ`f=#\<m2b}b0p\RA:ld7mc孶.xN]ufk[iG:پRZ<3ז%jkE8?'|'ټWK R{R~Ё+Q)$SBH1$ʒ`S2~Ps=PZ@DY ޅ:Sf=cE}(v-SQqҎ7C./W8M'jRR_g+癴m>zO{oH+*HQۨX[OKQ92JBX#aIy4}& %Z"'4!P8[|\IKP87kÄG]9BI1Q^a"^{繾A+T#UP8Uהo'uphb8\:aD77VZ'ި`DM!0t.6N*)hT1TpYZ b?)EJMxy/땶$hͭK{h5*PKܪ./B [zY\!yCUj}fն5%eg.⸧fyK&vnnkNqQಁs"v>U_z}CkOM~A{u a9ݒϮu&5ߦ!cߗhh6u͚!"yi2KVkzuY?|p晻 AѡE0T1mՠ*pwon޺?Fiz.ad⹭9+dX5%jΫ-XbE>q$( Tu0c(.C!CdG臏OO=Hv翚kg,/3% :%N6 ,vZa 9 E"do7 Q&U\ZU<_,/FfJ7UiYu"wy ȥ.(FE%rJ$W' 2ԋ$?X{B] ZLoSO:`8ӶfoƐ։Uܩ{V1ܖ!:~*?7LZƹ4:T}<>3YXi4i4G:@òvGB5߂GŐ IISu H (ZTcmBLx|Y_aRE#gGI:eeIvZOBXg}k .?\d 3,sJ̃F3gH"-32]ѿ0M=t) L{*6UDܽG.( & $b \u߀KuI%š3tXբ"ώlQ/}׼E`JRu"$v_3jkz5+iJ.%A]wH* ,O$h~w9kzJ%H/06nJwҮYS8A~wɊ v,?n=ZhZi]<+r63+q؟w\3X!|k?uuċ X/aX-\ZVe>Hy^\N\P=ވjR23fJ}2JIj T՝_Ru|;t'!RY,%~(jOdž!/eߟ\H#鿄7q= F 48 $u:E{#UYiȡš5*3҃+;a"BB&^]F$ĂX!3"Iee)]3- f\9;~Ô@PaoUBf/Ҟ^+CDE$*G?7с~"Ckq~鿸\̶R,Ϛ((g1fӫZ'z(EE.o[<낝/˴#CRGѡzQp=n#N&jWW rKwڐpQHm8+?h6 #YyVEfUogtkJ>z5FvԾ gؠ|"|yԾ|=BnCPQ|D]K>9sBƁg=ygHlvJ0|֎}5b{\ւT&)>rS1I.nM#۩uKfJ=ۚ$vqBr#Ƣ E: ɳO$9K`m*(;\*d)&ͪf }s^tK}~?bP\$Wg|ꭗ67Whzϵ?"Ŷ]15n8dٷ6˲ D0Xd`.c_ǒˠY#cLgiLyԝ8<8oj!XC}=)h?ϝxO\ l/-!:IDS1b/CqnY$P#fq#; @n[MTɫ4T:Hjc:*躽MOsRzuu[}:l5v)Ͱ".qBp@WPXG)K$䥢iI8KBo\e La7r9N}vmވ*vec*5]1{[, U/c@Ȕ(::$Us/MYFh[}gȁ],fv?C%"˙mv5j5vR[C/G2xZ3)o~2O0T%X7hlԥpY3x<iJӥ??LXx(.YXaFtBZB.ӠR&h7с$P/gLVP\WQFY\6-QkcmAi +rλŹ%mQyw"Ho۝fIaKB '-Ded,\eD\4r =ƟՂج,u؝Β{|8͗%΄7I~Rƴq 3Qw ;5E~\RCD$Qm$/}n盛M\#J$ݎ=rLY,(/¥[%`%mA,m2< ` >ΰM4I#/RE4a_;_DI&[!0t;;;Đ1ΤuC 1'}B46bs焼l48JaLY a+|+|הE Ar.Aބl4Cv) ~`*Y%#d\0vRr ui4m=Nvu^ٙ;Of36^*p%KduOnLw.HΫV>Rl<AwE#Nta  PAT,cfQm;9@l%ke-/:e5$PB_fŅDs۹LmK^.W>NQ ]f+Ye&#'w.*spA FVT M0k;MC&{h\??'{1<ڽv{cxy%\Q_ghdUxВ\/>΢C? {{ lV ~?Z%iS6/n;o+e9 Mkou_w~1'zMȥ>3R\ eCAtPC2Qx<gN@?{i+և~L@C[g(&$`Q&k=K ԉ\k1AОʾ !]*+C} "*3)fj0E8CO`^(F8a2e.eX"pa[E.˥<3E`rh1ņY y4K{Dž(H(#@U\ R߹tLӫϦ<{A"Ǻ#.1ֹaC /Kp4=(Yy`:*y(ILX(>2ĝ"&7Ve~*ʋ-B!)c }qJ3Fθ[_;Yo[3{#NmRn,WqHCFʣ Ċ?kTmmLvdċ4dyx&sT6Q]Ϝy:tI,>IZ`ĐC_jA` "Ș T$2!D3[PHG?gT\wKY.K!!-Q6)GT0)ݓ,{xw֦R8[r?T#+F 6\-.'p {sv#IP ǵ e{fƙ߹}VQX=~קEgT3R$8*DOuD;>WFq(赛TH6-"oeś)~aN cu;w8MKTx6]sF;+ABCm,_uԙ~ϲ3鿹Qi@kvμakz4hfk j.~n,ΒKTM:H0Bum@`QK5APkO 1f*5<dL0210!ġC{KDvs.>/ (1IRWz Q{=T=_8.TXNH(폟3u3Q)#d vc/$a9mu!+_nLc}̏'xG}}OS2n,X"r &8AL.i%<(}?YAeBs^=eJߨsa⽧ry7AZ$xڭx[%C |a+is!ʞ/?%Ʌ~ǐF[ 6[JGBm^Xm28YsXtP/4dqНP?LU"@@_'! ] (5@5t o-bȏ?zqRīF]\{@$/P"k \qWߛQW K*̚iy#O++ Gݺt[zg(Fn3vFt.}._J#㭊3A)HrY)ⷛB&TfVj>Ʌ~c{c؟޸ Vwƌn敐h BLRl|b%9Ǭ|!"ZAx;/#،_P8 >EDB& YHIhhBZIT !P V<`:Gd^ `1Y *Ka梓, I-0ઽ@mZ_P*noU|qG1d0Mi}5"08K.bx,"04u?z q> ! b\q'IإtH@dAfLCC X$Ō7sQq$*@oWx!(o#]>dPfI ÅHu>rpr\2:=<4}{7wqw/w_g3[w>sup eӁw2'W5$JpduXeQ$|{.GWtcA{ɸcM܁Tg IcHe#O& b1i0$LnRl0E%~5T[11$Kt]ޓ0&zcFsAc ;sL0t!ayuV'}( @R鞸nxBG+sE{݀~ zI1-=J #Kz X$(ZvCQG "qz8l[f^W7f^vcu%wgiu8iiEi%a|GQJ;,8%anYͷLwi(LA^quڇ YZв+$j<0#+|R㌳ #PY346p® :t tPgLo?+ƭWAjadyXcu R)2E=DÒޞ[TgI\.D~q:KיJež!=!2Zfzޗb$2A]'J<, {05a+d1_h~fNz}H)c2klFƝmJNfiϮw*e:io>[ĭ%`Ǿ,NL|}Hl}H19wN`a-R0 D:9vCBd pɆUaWt3F}dɥ5Ti6IJZ͚+*qf(;X1 *It 2u'*CJDGW YDOЇZg!)] ixZRumu>t*F/}3&s%P ʅ%JiǶq‡Xo.%-K$fhy0ZxC6,XEF:BRTdlH[_Qo|&Ha8gwCf=,OM3{˅{v™ Ϊ6nuy SxVA% 34/~!WrK2bXH+dQvGPV31E Va{LRn)ob$Y娗1<5c1BOV)߄N UA6k_ܬBHEu1쌀Dz0Fe%^7Xt@|"oܥdHgT]E`ϰĸy 9/d }i8y?cOEM=#pL׉!(!t!@*dg!gx0v"L!Gs{y!ά4 P^d\F??~'V3\5(ZSEsa0~|?z<@ Թ}BH8[D6#5ZzY]IHA0o! ~^q"AԍyͱA@R4V'_*$2!"RobAD/g|J`p5ңHƗYԤH6k;33bŌhG,@T%L"ž3f2,[ Ti#3tt5mk>]]hӐQo#-{Y՗z%ɜ5L鐱e7k} Pʄ/:!-^2MHO2nYiQDA4}m1Y[W6\a6u=1> J,(~jUƖB،FF:DE$ыJivSɡ"ͅXdAr#yT8UD ƆZ 2@iˉ=BĥĒ K-:cfјwԬ:H$SV+:b!"%,фBE|-[Eg[!ϕOgָ[j ֶo;4ՃuaN W7qd䳮vPfWh6W ?+AlLx13Ѿ& ydQۓ!b~,a}^fvmH%+8'cZD>jݸ2Bȣͮ+4I zF3CZH6gRwȆ]oq=_b)jG%ƣ K| *(?yo?KVՁiߦPL2yu& Լ-=1\DY#q5B}6D|V$"LڦcNt*LI$%²Jx~Kk^؝͚a>l 6kpIh BbMa=)*X4AruU˓@p8)1AϜSQ,Ru@h}eI7 c++Z ~ANx>냅WI{peGtY= ;!B\"4s`oMu6}~WG=f>T2yHt#1KnT8R.}UlaƉ|uN(!]8Gt&|~6nz7~IV!cgC#D־2amWnVVA2>%k}Y!5o~+|Bna-ţ0D~C8ӨP[ ~65ku&xmP_ ?*ף@`t2d6:wUĚXR#ļ n`l~}4ngk-A6`@ۢtZ݈ L=ϚZ5ңQPxxF?ZȌؕP91AxS8M rF#nrY K/N߾D`O~΅>!z:SPZc~1Wi`^$*Ч3ƟMcA?k6 |fJKݓUgɗ! FsdHs ?慤f\Kͱo_ܶ߅BHu0\ԋ%bH)qp%2N6$ɽA803K @mH C[2 SJc<]R{/z].Aze1HB!Wה— ? N吝q!y ߿H`(9H\;&}x2(U!_ze^'rX6¹Cy3JP"k|S3{)["k^ځ QzƒH;A%%!|IƗP"kPq'Ջ-gf:a "!27{*~ɥx=,;|![ow/Kv9?hUP[ Tbو>wY2H4smi\nܛE`u!5`.0($~nHO0قY+ӫ7S̝_:uQ5m'D !CIZ02oDJXa$Cl -n&W|K_^P$طH-<=Bjy88OaL;CeB &G9M*j.pVAQKT Xy^Ohu7 <Zl\=Nyz[(2Bj._"k=%zY"BlI'ݪ$m7^d/|qsA_vlyϴd$Bjچp lO2{M[ s-/mQ?(&j?BF !5dRWkf9o6o+/!<-[vk5=}#k^[as.=MygfHlm? ͔'A[9j˼44JbH:cBdͺu v^!d̐#&*m&]szb/S0 dsl:G]X#Oϵ9~BksE(X]yCLl;ΤXy'<i2L͞ ̟9R;$< nXh2Cdݼ Ry#9@ =p<BF%_B){؜ BkDl/8׺&[[(^^1^!6 #bov±GD;jO>ЯgϽCՈ)Đ/vƘ;s>'}]gl8pp`lGN^żUXI&\ I^9h̸Ծ Vh;2ݽF!#N~0.nCD*ynhlLu)mj?۾-xxV]0V4֨rR0|,jO&oJ`wUA3[ꀸu]Fg<S]ƶ"!F=c 2BFfUʟh $zj41Y }, },6(l,dZ:m 0@qM~z!.M+Ҳu;&$`$&+S\:RcADͻكsjϫiŔWW KG!X!dLX纪3Ϛ*|1"aQ|f4cY.#l+z,(?Sot0F1B_uE8( 8t6v"&(k 2Vfˇý#*( |#1$|<^1Ë!~%±?Mw ಱ~eUsmʒI謒o^'p}ȷ7nZ 槻Չ8 1dg^]`˪{MKО8 DGƲHIV}_GOAgf=e3]PۦgA&!cOϵxpUp˙,?;l6EMRUBJY#j)%f>ƫ{6Ìgy_V"F[-pV-mOd>]Z~{C&!cl3yxMˋqI,/#2(G%gSXÊ >Q \(ȥWBjp.$fє\iQ]Ų^Nt)A5o68B3O(h25ءGUq)mmuIu x L:;DӖ/U.seE/85{D ̣  f$Żd^t@B&B}?_{ѡNþfŲUn<"?%~H/:| B3tVBtZg( cV>M]D|"wq/Ÿ'svB@O2J("LJQ \+"(CMMʲ6Q `OIENDB`apache-datafusion-python-54.0.0/docs/source/_static/images/original_dark.svg000066400000000000000000000215031521727353300271550ustar00rootroot00000000000000 apache-datafusion-python-54.0.0/docs/source/_static/theme_overrides.css000066400000000000000000000057611521727353300262700ustar00rootroot00000000000000/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * software distributed under the License is distributed on an * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * KIND, either express or implied. See the License for the * specific language governing permissions and limitations * under the License. */ /* Customizing with theme CSS variables */ :root { --pst-color-link-hover: 215, 70, 51; --pst-color-headerlink: 215, 70, 51; /* Softer blue from bootstrap's default info color */ --pst-color-info: 23, 162, 184; } code { color: rgb(215, 70, 51); } html[data-theme="dark"] code { color: rgb(255, 138, 117); } .footer { text-align: center; } /* Bootstrap "table-striped" applied globally so individual tables in user-guide pages don't need ":class: table-striped" added one by one. */ .table tbody tr:nth-of-type(odd) { background-color: rgba(0, 0, 0, 0.05); } html[data-theme="dark"] .table tbody tr:nth-of-type(odd) { background-color: rgba(255, 255, 255, 0.05); } /* Fix table text wrapping in RTD theme, * see https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html */ @media screen { table.docutils td { /* !important prevents the common CSS stylesheets from overriding this as on RTD they are loaded after this stylesheet */ white-space: normal !important; } } /* Hideable right-hand "On this page" sidebar. * toc-toggle.js adds the button and toggles `pst-secondary-hidden` on ; * hiding the sidebar lets the flex article container reclaim the width. */ body.pst-secondary-hidden .bd-sidebar-secondary { display: none; } /* Let the article use the freed space rather than just re-centering. */ body.pst-secondary-hidden .bd-article-container { max-width: none; } /* Floating toggle button, pinned to the top-right under the navbar. */ #pst-secondary-toggle { position: fixed; top: 4.5rem; right: 0.75rem; z-index: 1020; display: flex; align-items: center; justify-content: center; width: 2rem; height: 2rem; padding: 0; border: 1px solid var(--pst-color-border, #ccc); border-radius: 0.25rem; background-color: var(--pst-color-surface, #fff); color: var(--pst-color-text-base, #333); cursor: pointer; } #pst-secondary-toggle:hover { color: rgb(var(--pst-color-link-hover)); } /* The toggle is only meaningful where the sidebar is shown (wide screens); * below the theme's lg breakpoint the sidebar is already collapsed away. */ @media (max-width: 959.98px) { #pst-secondary-toggle { display: none; } } apache-datafusion-python-54.0.0/docs/source/_static/toc-toggle.js000066400000000000000000000044061521727353300247670ustar00rootroot00000000000000/* * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements. See the NOTICE file * distributed with this work for additional information * regarding copyright ownership. The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at * * http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, * software distributed under the License is distributed on an * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY * KIND, either express or implied. See the License for the * specific language governing permissions and limitations * under the License. */ /* Adds a button that hides the right-hand "On this page" sidebar so the * article can use the full page width. The choice is remembered across * pages via localStorage. */ (function () { "use strict"; var KEY = "pst-secondary-hidden"; function apply(hidden, btn) { document.body.classList.toggle("pst-secondary-hidden", hidden); if (btn) { btn.setAttribute("aria-pressed", String(hidden)); btn.title = hidden ? "Show page contents" : "Hide page contents"; } } document.addEventListener("DOMContentLoaded", function () { // Only offer the toggle on pages that actually have the sidebar. if (!document.querySelector(".bd-sidebar-secondary")) { return; } var btn = document.createElement("button"); btn.id = "pst-secondary-toggle"; btn.type = "button"; btn.setAttribute("aria-label", "Toggle page contents sidebar"); btn.innerHTML = ''; document.body.appendChild(btn); btn.addEventListener("click", function () { var hidden = !document.body.classList.contains("pst-secondary-hidden"); try { localStorage.setItem(KEY, hidden ? "1" : "0"); } catch (e) { /* localStorage may be unavailable; toggle still works for this page. */ } apply(hidden, btn); }); var stored = null; try { stored = localStorage.getItem(KEY); } catch (e) { /* ignore */ } apply(stored === "1", btn); }); })(); apache-datafusion-python-54.0.0/docs/source/_templates/000077500000000000000000000000001521727353300230705ustar00rootroot00000000000000apache-datafusion-python-54.0.0/docs/source/_templates/layout.html000066400000000000000000000012641521727353300252760ustar00rootroot00000000000000{% extends "pydata_sphinx_theme/layout.html" %} {% block footer %}

{% endblock %} apache-datafusion-python-54.0.0/docs/source/_templates/sidebar-globaltoc.html000066400000000000000000000025751521727353300273440ustar00rootroot00000000000000{# Renders the global document toctree on every page (including the landing page) with pydata-sphinx-theme's collapsible chevrons. The stock sidebar-nav-bs.html starts at the current section and is stripped from the sidebar list by suppress_sidebar_toctree() on the root page (no parent section). Using generate_toctree_html with startdepth=0 renders the whole tree from root with the bootstrap classes the theme's JS uses for expand/collapse toggles. Naming the template "sidebar-globaltoc" sidesteps the suppress filter, which matches on "sidebar-nav-bs.html" specifically. #} apache-datafusion-python-54.0.0/docs/source/conf.py000066400000000000000000000176441521727353300222460ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Documentation generation.""" # Configuration file for the Sphinx documentation builder. # # This file only contains a selection of the most common options. For a full # list see the documentation: # https://www.sphinx-doc.org/en/master/usage/configuration.html # -- Path setup -------------------------------------------------------------- # If extensions (or modules to document with autodoc) are in another directory, # add these directories to sys.path here. If the directory is relative to the # documentation root, use os.path.abspath to make it absolute, like shown here. # # import os # import sys # sys.path.insert(0, os.path.abspath('.')) # -- Project information ----------------------------------------------------- project = "Apache DataFusion in Python" copyright = "2019-2026, Apache Software Foundation" author = "Apache Software Foundation" # -- General configuration --------------------------------------------------- # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. extensions = [ "sphinx.ext.mathjax", "sphinx.ext.napoleon", # myst_nb is a superset of myst_parser: it provides the MyST markdown # parser plus executable `{code-cell}` notebook directives. Do NOT also # list "myst_parser" — myst_nb activates it internally and listing both # raises an extension conflict. "myst_nb", "autoapi.extension", ] # NOTE: .rst stays alongside .md because sphinx-autoapi generates RST # under autoapi/ and Sphinx needs the suffix to parse it. The human- # authored docs are all MyST .md now. ".md" is routed through myst-nb so # pages carrying jupytext/kernelspec front matter execute their # `{code-cell}` blocks; pages without that front matter render as plain # MyST markdown. The ".rst" entry is only for the autoapi build artifacts. source_suffix = { ".rst": "restructuredtext", ".md": "myst-nb", } # Execute notebook code cells at build time and fail the build if any cell # raises — this replaces the old IPython sphinx directive, whose executed # examples are now `{code-cell}` blocks. "force" re-executes every build so # stale cached output can never ship. nb_execution_mode = "force" nb_execution_timeout = 120 nb_execution_raise_on_error = True # Prefer the plain-text repr of a cell's last expression over its rich # `_repr_html_`. A DataFrame's HTML repr is a self-contained widget (inline # styles + an injected """ html = [script] if self.custom_css: html.append(f"") return html def _build_table_container_start(self) -> list[str]: """Build the opening tags for the table container.""" html = [] html.append( f'
' ) html.append('') return html def _build_table_header(self, schema: Any) -> list[str]: """Build the HTML table header with column names.""" html = [] html.append("") html.append("") for field in schema: if self._custom_header_builder: html.append(self._custom_header_builder(field)) else: html.append( f"" ) html.append("") html.append("") return html def _build_table_body(self, batches: list, table_uuid: str) -> list[str]: """Build the HTML table body with data rows.""" html = [] html.append("") row_count = 0 for batch in batches: for row_idx in range(batch.num_rows): row_count += 1 html.append("") for col_idx, column in enumerate(batch.columns): # Get the raw value from the column raw_value = self._get_cell_value(column, row_idx) # Always check for type formatters first to format the value formatted_value = self._format_cell_value(raw_value) # Then apply either custom cell builder or standard cell formatting if self._custom_cell_builder: # Pass both the raw value and formatted value to let the # builder decide cell_html = self._custom_cell_builder( raw_value, row_count, col_idx, table_uuid ) html.append(cell_html) else: # Standard cell formatting with formatted value if ( len(str(raw_value)) > self.max_cell_length and self.enable_cell_expansion ): cell_html = self._build_expandable_cell( formatted_value, row_count, col_idx, table_uuid ) else: cell_html = self._build_regular_cell(formatted_value) html.append(cell_html) html.append("") html.append("") return html def _get_cell_value(self, column: Any, row_idx: int) -> Any: """Extract a cell value from a column. Args: column: Arrow array row_idx: Row index Returns: The raw cell value """ try: value = column[row_idx] if hasattr(value, "as_py"): return value.as_py() except (AttributeError, TypeError): pass else: return value def _format_cell_value(self, value: Any) -> str: """Format a cell value for display. Uses registered type formatters if available. Args: value: The cell value to format Returns: Formatted cell value as string """ # Check for custom type formatters for type_cls, formatter in self._type_formatters.items(): if isinstance(value, type_cls): return formatter(value) # If no formatter matched, return string representation return str(value) def _build_expandable_cell( self, formatted_value: str, row_count: int, col_idx: int, table_uuid: str ) -> str: """Build an expandable cell for long content.""" short_value = str(formatted_value)[: self.max_cell_length] return ( f"" ) def _build_regular_cell(self, formatted_value: str) -> str: """Build a regular table cell.""" return ( f"" ) def _build_html_footer(self, has_more: bool) -> list[str]: """Build the HTML footer with JavaScript and messages.""" html = [] # Add JavaScript for interactivity only if cell expansion is enabled # and we're not using the shared styles approach if self.enable_cell_expansion and not self.use_shared_styles: html.append(self._get_javascript()) # Add truncation message if needed if has_more and self.show_truncation_message: html.append("
Data truncated due to size.
") return html def _get_default_css(self) -> str: """Get default CSS styles for the HTML table.""" return """ .expandable-container { display: inline-block; max-width: 200px; } .expandable { white-space: nowrap; overflow: hidden; text-overflow: ellipsis; display: block; } .full-text { display: none; white-space: normal; } .expand-btn { cursor: pointer; color: blue; text-decoration: underline; border: none; background: none; font-size: inherit; display: block; margin-top: 5px; } """ def _get_javascript(self) -> str: """Get JavaScript code for interactive elements.""" return """ """ class FormatterManager: """Manager class for the global DataFrame HTML formatter instance.""" _default_formatter: DataFrameHtmlFormatter = DataFrameHtmlFormatter() @classmethod def set_formatter(cls, formatter: DataFrameHtmlFormatter) -> None: """Set the global DataFrame HTML formatter. Args: formatter: The formatter instance to use globally """ cls._default_formatter = formatter _refresh_formatter_reference() @classmethod def get_formatter(cls) -> DataFrameHtmlFormatter: """Get the current global DataFrame HTML formatter. Returns: The global HTML formatter instance """ return cls._default_formatter def get_formatter() -> DataFrameHtmlFormatter: """Get the current global DataFrame HTML formatter. This function is used by the DataFrame._repr_html_ implementation to access the shared formatter instance. It can also be used directly when custom HTML rendering is needed. Returns: The global HTML formatter instance Example: >>> from datafusion.dataframe_formatter import get_formatter >>> formatter = get_formatter() >>> formatter.max_cell_length = 50 # Increase cell length """ return FormatterManager.get_formatter() def set_formatter(formatter: DataFrameHtmlFormatter) -> None: """Set the global DataFrame HTML formatter. Args: formatter: The formatter instance to use globally Example: >>> from datafusion.dataframe_formatter import get_formatter, set_formatter >>> custom_formatter = DataFrameHtmlFormatter(max_cell_length=100) >>> set_formatter(custom_formatter) """ FormatterManager.set_formatter(formatter) def configure_formatter(**kwargs: Any) -> None: """Configure the global DataFrame HTML formatter. This function creates a new formatter with the provided configuration and sets it as the global formatter for all DataFrames. Args: **kwargs: Formatter configuration parameters like max_cell_length, max_width, max_height, enable_cell_expansion, etc. Raises: ValueError: If any invalid parameters are provided Example: >>> from datafusion.dataframe_formatter import configure_formatter >>> configure_formatter( ... max_cell_length=50, ... max_height=500, ... enable_cell_expansion=True, ... use_shared_styles=True ... ) """ # Valid parameters accepted by DataFrameHtmlFormatter valid_params = { "max_cell_length", "max_width", "max_height", "max_memory_bytes", "min_rows", "max_rows", "repr_rows", "enable_cell_expansion", "custom_css", "show_truncation_message", "style_provider", "use_shared_styles", } # Check for invalid parameters invalid_params = set(kwargs) - valid_params if invalid_params: msg = ( f"Invalid formatter parameters: {', '.join(invalid_params)}. " f"Valid parameters are: {', '.join(valid_params)}" ) raise ValueError(msg) # Create and set formatter with validated parameters set_formatter(DataFrameHtmlFormatter(**kwargs)) def reset_formatter() -> None: """Reset the global DataFrame HTML formatter to default settings. This function creates a new formatter with default configuration and sets it as the global formatter for all DataFrames. Example: >>> from datafusion.dataframe_formatter import reset_formatter >>> reset_formatter() # Reset formatter to default settings """ formatter = DataFrameHtmlFormatter() set_formatter(formatter) def _refresh_formatter_reference() -> None: """Refresh formatter reference in any modules using it. This helps ensure that changes to the formatter are reflected in existing DataFrames that might be caching the formatter reference. """ # This is a no-op but signals modules to refresh their reference apache-datafusion-python-54.0.0/python/datafusion/expr.py000066400000000000000000001725571521727353300235320ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """:py:class:`Expr` — the logical expression type used to build DataFusion queries. An :py:class:`Expr` represents a computation over columns or literals: a column reference (``col("a")``), a literal (``lit(5)``), an operator combination (``col("a") + lit(1)``), or the output of a function from :py:mod:`datafusion.functions`. Expressions are passed to :py:class:`~datafusion.dataframe.DataFrame` methods such as :py:meth:`~datafusion.dataframe.DataFrame.select`, :py:meth:`~datafusion.dataframe.DataFrame.filter`, :py:meth:`~datafusion.dataframe.DataFrame.aggregate`, and :py:meth:`~datafusion.dataframe.DataFrame.sort`. Convenience constructors are re-exported at the package level: :py:func:`datafusion.col` / :py:func:`datafusion.column` for column references and :py:func:`datafusion.lit` / :py:func:`datafusion.literal` for scalar literals. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> df.select((col("a") * lit(10)).alias("ten_a")).to_pydict() {'ten_a': [10, 20, 30]} See :ref:`expressions` in the online documentation for details on available operators and helpers. """ # ruff: noqa: PLC0415 from __future__ import annotations from collections.abc import Callable, Iterable, Sequence from typing import TYPE_CHECKING, Any, ClassVar import pyarrow as pa from ._internal import expr as expr_internal from ._internal import functions as functions_internal from .ipc import get_sender_ctx if TYPE_CHECKING: from collections.abc import Sequence from datafusion.common import ( # type: ignore[import] DataTypeMap, NullTreatment, RexType, ) from datafusion.context import SessionContext from datafusion.plan import LogicalPlan # Standard error message for invalid expression types # Mention both alias forms of column and literal helpers EXPR_TYPE_ERROR = "Use col()/column() or lit()/literal() to construct expressions" # The following are imported from the internal representation. We may choose to # give these all proper wrappers, or to simply leave as is. These were added # in order to support passing the `test_imports` unit test. # Tim Saucer note: It is not clear to me what the use case is for exposing # these definitions to the end user. Alias = expr_internal.Alias Analyze = expr_internal.Analyze Aggregate = expr_internal.Aggregate AggregateFunction = expr_internal.AggregateFunction Between = expr_internal.Between BinaryExpr = expr_internal.BinaryExpr Case = expr_internal.Case Cast = expr_internal.Cast Column = expr_internal.Column CopyTo = expr_internal.CopyTo CreateCatalog = expr_internal.CreateCatalog CreateCatalogSchema = expr_internal.CreateCatalogSchema CreateExternalTable = expr_internal.CreateExternalTable CreateFunction = expr_internal.CreateFunction CreateFunctionBody = expr_internal.CreateFunctionBody CreateIndex = expr_internal.CreateIndex CreateMemoryTable = expr_internal.CreateMemoryTable CreateView = expr_internal.CreateView Deallocate = expr_internal.Deallocate DescribeTable = expr_internal.DescribeTable Distinct = expr_internal.Distinct DmlStatement = expr_internal.DmlStatement DropCatalogSchema = expr_internal.DropCatalogSchema DropFunction = expr_internal.DropFunction DropTable = expr_internal.DropTable DropView = expr_internal.DropView EmptyRelation = expr_internal.EmptyRelation Execute = expr_internal.Execute Exists = expr_internal.Exists Explain = expr_internal.Explain Extension = expr_internal.Extension FileType = expr_internal.FileType Filter = expr_internal.Filter Join = expr_internal.Join ILike = expr_internal.ILike InList = expr_internal.InList InSubquery = expr_internal.InSubquery IsFalse = expr_internal.IsFalse IsNotTrue = expr_internal.IsNotTrue IsNull = expr_internal.IsNull IsTrue = expr_internal.IsTrue IsUnknown = expr_internal.IsUnknown IsNotFalse = expr_internal.IsNotFalse IsNotNull = expr_internal.IsNotNull IsNotUnknown = expr_internal.IsNotUnknown JoinConstraint = expr_internal.JoinConstraint JoinType = expr_internal.JoinType Like = expr_internal.Like Limit = expr_internal.Limit Literal = expr_internal.Literal Negative = expr_internal.Negative Not = expr_internal.Not OperateFunctionArg = expr_internal.OperateFunctionArg Partitioning = expr_internal.Partitioning Placeholder = expr_internal.Placeholder Prepare = expr_internal.Prepare Projection = expr_internal.Projection RecursiveQuery = expr_internal.RecursiveQuery Repartition = expr_internal.Repartition ScalarSubquery = expr_internal.ScalarSubquery ScalarVariable = expr_internal.ScalarVariable SetVariable = expr_internal.SetVariable SimilarTo = expr_internal.SimilarTo Sort = expr_internal.Sort Subquery = expr_internal.Subquery SubqueryAlias = expr_internal.SubqueryAlias TableScan = expr_internal.TableScan TransactionAccessMode = expr_internal.TransactionAccessMode TransactionConclusion = expr_internal.TransactionConclusion TransactionEnd = expr_internal.TransactionEnd TransactionIsolationLevel = expr_internal.TransactionIsolationLevel TransactionStart = expr_internal.TransactionStart TryCast = expr_internal.TryCast Union = expr_internal.Union HigherOrderFunction = expr_internal.HigherOrderFunction Lambda = expr_internal.Lambda LambdaVariable = expr_internal.LambdaVariable Unnest = expr_internal.Unnest UnnestExpr = expr_internal.UnnestExpr Values = expr_internal.Values WindowExpr = expr_internal.WindowExpr __all__ = [ "EXPR_TYPE_ERROR", "Aggregate", "AggregateFunction", "Alias", "Analyze", "Between", "BinaryExpr", "Case", "CaseBuilder", "Cast", "Column", "CopyTo", "CreateCatalog", "CreateCatalogSchema", "CreateExternalTable", "CreateFunction", "CreateFunctionBody", "CreateIndex", "CreateMemoryTable", "CreateView", "Deallocate", "DescribeTable", "Distinct", "DmlStatement", "DropCatalogSchema", "DropFunction", "DropTable", "DropView", "EmptyRelation", "Execute", "Exists", "Explain", "Expr", "Extension", "FileType", "Filter", "GroupingSet", "HigherOrderFunction", "ILike", "InList", "InSubquery", "IsFalse", "IsNotFalse", "IsNotNull", "IsNotTrue", "IsNotUnknown", "IsNull", "IsTrue", "IsUnknown", "Join", "JoinConstraint", "JoinType", "Lambda", "LambdaVariable", "Like", "Limit", "Literal", "Literal", "Negative", "Not", "OperateFunctionArg", "Partitioning", "Placeholder", "Prepare", "Projection", "RecursiveQuery", "Repartition", "ScalarSubquery", "ScalarVariable", "SetVariable", "SimilarTo", "Sort", "SortExpr", "SortKey", "Subquery", "SubqueryAlias", "TableScan", "TransactionAccessMode", "TransactionConclusion", "TransactionEnd", "TransactionIsolationLevel", "TransactionStart", "TryCast", "Union", "Unnest", "UnnestExpr", "Values", "Window", "WindowExpr", "WindowFrame", "WindowFrameBound", "coerce_to_expr", "coerce_to_expr_list", "coerce_to_expr_or_none", "ensure_expr", "ensure_expr_list", ] def ensure_expr(value: Expr | Any) -> expr_internal.Expr: """Return the internal expression from ``Expr`` or raise ``TypeError``. This helper rejects plain strings and other non-:class:`Expr` values so higher level APIs consistently require explicit :func:`~datafusion.col` or :func:`~datafusion.lit` expressions. See Also: :func:`coerce_to_expr` — the opposite behavior: *wraps* non-``Expr`` values as literals instead of rejecting them. Args: value: Candidate expression or other object. Returns: The internal expression representation. Raises: TypeError: If ``value`` is not an instance of :class:`Expr`. """ if not isinstance(value, Expr): raise TypeError(EXPR_TYPE_ERROR) return value.expr def ensure_expr_list( exprs: Iterable[Expr | Iterable[Expr]], ) -> list[expr_internal.Expr]: """Flatten an iterable of expressions, validating each via ``ensure_expr``. Args: exprs: Possibly nested iterable containing expressions. Returns: A flat list of raw expressions. Raises: TypeError: If any item is not an instance of :class:`Expr`. """ def _iter( items: Iterable[Expr | Iterable[Expr]], ) -> Iterable[expr_internal.Expr]: for expr in items: if isinstance(expr, Iterable) and not isinstance( expr, Expr | str | bytes | bytearray ): # Treat string-like objects as atomic to surface standard errors yield from _iter(expr) else: yield ensure_expr(expr) return list(_iter(exprs)) def coerce_to_expr(value: Any) -> Expr: """Coerce a native Python value to an ``Expr`` literal, passing ``Expr`` through. This is the complement of :func:`ensure_expr`: where ``ensure_expr`` *rejects* non-``Expr`` values, ``coerce_to_expr`` *wraps* them via :meth:`Expr.literal` so that functions can accept native Python types (``int``, ``float``, ``str``, ``bool``, etc.) alongside ``Expr``. Args: value: An ``Expr`` instance (returned as-is) or a Python literal to wrap. Returns: An ``Expr`` representing the value. """ if isinstance(value, Expr): return value return Expr.literal(value) def coerce_to_expr_or_none(value: Any | None) -> Expr | None: """Coerce a value to ``Expr`` or pass ``None`` through unchanged. Same as :func:`coerce_to_expr` but accepts ``None`` for optional parameters. Args: value: An ``Expr`` instance, a Python literal to wrap, or ``None``. Returns: An ``Expr`` representing the value, or ``None``. """ if value is None: return None return coerce_to_expr(value) def coerce_to_expr_list(values: Iterable[Any]) -> list[Expr]: """Coerce each item in an iterable to ``Expr`` via :func:`coerce_to_expr`. Args: values: Iterable of ``Expr`` instances or Python literals to wrap. Returns: A list of ``Expr`` instances. """ return [coerce_to_expr(value) for value in values] def _to_raw_expr(value: Expr | str) -> expr_internal.Expr: """Convert a Python expression or column name to its raw variant. Args: value: Candidate expression or column name. Returns: The internal :class:`~datafusion._internal.expr.Expr` representation. Raises: TypeError: If ``value`` is neither an :class:`Expr` nor ``str``. """ if isinstance(value, str): return Expr.column(value).expr if isinstance(value, Expr): return value.expr error = ( "Expected Expr or column name, found:" f" {type(value).__name__}. {EXPR_TYPE_ERROR}." ) raise TypeError(error) def expr_list_to_raw_expr_list( expr_list: list[Expr] | Expr | None, ) -> list[expr_internal.Expr] | None: """Convert a sequence of expressions or column names to raw expressions.""" if isinstance(expr_list, Expr | str): expr_list = [expr_list] if expr_list is None: return None return [_to_raw_expr(e) for e in expr_list] def sort_or_default(e: Expr | SortExpr) -> expr_internal.SortExpr: """Helper function to return a default Sort if an Expr is provided.""" if isinstance(e, SortExpr): return e.raw_sort return SortExpr(e, ascending=True, nulls_first=True).raw_sort def sort_list_to_raw_sort_list( sort_list: Sequence[SortKey] | SortKey | None, ) -> list[expr_internal.SortExpr] | None: """Helper function to return an optional sort list to raw variant.""" if isinstance(sort_list, Expr | SortExpr | str): sort_list = [sort_list] if sort_list is None: return None raw_sort_list = [] for item in sort_list: if isinstance(item, SortExpr): raw_sort_list.append(sort_or_default(item)) else: raw_expr = _to_raw_expr(item) # may raise ``TypeError`` raw_sort_list.append(sort_or_default(Expr(raw_expr))) return raw_sort_list class Expr: # noqa: PLW1641 """Expression object. Expressions are one of the core concepts in DataFusion. See :ref:`Expressions` in the online documentation for more information. """ def __init__(self, expr: expr_internal.RawExpr) -> None: """This constructor should not be called by the end user.""" self.expr = expr def to_variant(self) -> Any: """Convert this expression into a python object if possible.""" return self.expr.to_variant() def schema_name(self) -> str: """Returns the name of this expression as it should appear in a schema. This name will not include any CAST expressions. """ return self.expr.schema_name() def canonical_name(self) -> str: """Returns a complete string representation of this expression.""" return self.expr.canonical_name() def variant_name(self) -> str: """Returns the name of the Expr variant. Ex: ``IsNotNull``, ``Literal``, ``BinaryExpr``, etc """ return self.expr.variant_name() def to_bytes(self, ctx: SessionContext | None = None) -> bytes: """Serialize this expression to bytes for shipping to another process. Use this — or :func:`pickle.dumps` — to send an expression to a worker process for distributed evaluation. When ``ctx`` is supplied, encoding routes through that session's installed :class:`LogicalExtensionCodec` (so settings like :meth:`SessionContext.with_python_udf_inlining` take effect). When ``ctx`` is ``None``, the default codec is used (Python UDF inlining on, no user-installed extension codec). Built-in functions travel inside the returned bytes. Python UDFs (scalar, aggregate, window) also inline by default, so the worker does not need to pre-register them; when the encoding session has :meth:`SessionContext.with_python_udf_inlining` set to ``False``, Python UDFs travel by name only and must be registered on the worker. UDFs imported via the FFI capsule protocol always travel by name only and must be registered on the worker. .. warning:: Security Bytes returned here may embed a cloudpickled Python callable (when the expression carries a Python UDF). Reconstructing them via :meth:`from_bytes` or :func:`pickle.loads` executes arbitrary Python on the receiver. Only accept payloads from trusted sources. .. warning:: Portability cloudpickle serializes Python bytecode, which is **not stable across Python minor versions**. A payload produced on Python 3.11 will fail to load on Python 3.12. The wire format stamps the sender's ``(major, minor)``; :meth:`from_bytes` raises a :class:`ValueError` naming both versions on mismatch. cloudpickle captures the UDF callable **by value** — bytecode and closure cells inlined — but names the callable resolves via ``import`` are captured **by reference** (module path only) and must be importable on the receiver. **Self-contained — works anywhere:** .. code-block:: python # Lambda: bytecode captured inline udf(lambda x: x * 2, [pa.int64()], pa.int64(), volatility="immutable") # Locally-defined function: bytecode captured inline def double(x): return x * 2 udf(double, [pa.int64()], pa.int64(), volatility="immutable") # Closure over a local variable: value captured inline factor = 3 udf(lambda x: x * factor, [pa.int64()], pa.int64(), volatility="immutable") **Requires matching environment on receiver:** .. code-block:: python # Top-level import: `foo` must be installed on receiver from foo import double udf(double, [pa.int64()], pa.int64(), volatility="immutable") # Bound method of an imported class: same caveat from mylib import Transformer t = Transformer() udf(t.transform, [pa.int64()], pa.int64(), volatility="immutable") Examples: >>> from datafusion import col, lit >>> blob = (col("a") + lit(1)).to_bytes() >>> isinstance(blob, bytes) True """ ctx_arg = ctx.ctx if ctx is not None else None return self.expr.to_bytes(ctx_arg) @classmethod def from_bytes(cls, buf: bytes, ctx: SessionContext | None = None) -> Expr: """Reconstruct an expression from serialized bytes. Accepts output of :meth:`to_bytes` or :func:`pickle.dumps`. ``ctx`` is the :class:`SessionContext` used to resolve any function references that travel by name (e.g. FFI UDFs, or Python UDFs sent with inlining disabled via :meth:`SessionContext.with_python_udf_inlining`). When ``ctx`` is ``None`` the worker context installed via :func:`datafusion.ipc.set_worker_ctx` is consulted; if no worker context is installed, the global :class:`SessionContext` is used (sufficient for built-ins and Python UDFs, plus any UDFs registered on the global context). .. warning:: Security Decoding may invoke ``cloudpickle.loads`` on bytes embedded in the payload, which executes arbitrary Python code. Treat ``buf`` as code, not data — only decode bytes you produced yourself or received from a trusted sender. .. warning:: Portability cloudpickle payloads are **not portable across Python minor versions**. The wire format stamps the sender's ``(major, minor)``; if it does not match the current interpreter, this method raises :class:`ValueError` naming both versions. Modules the UDF imports must also be importable on the receiver — see :meth:`to_bytes` for by-value vs. by-reference details. Examples: >>> from datafusion import Expr, col, lit >>> blob = (col("a") + lit(1)).to_bytes() >>> Expr.from_bytes(blob).canonical_name() 'a + Int64(1)' """ from datafusion.ipc import _resolve_ctx resolved = _resolve_ctx(ctx) return cls(expr_internal.RawExpr.from_bytes(resolved.ctx, buf)) def __reduce__(self) -> tuple[Callable[[bytes], Expr], tuple[bytes]]: """Pickle protocol hook. Lets expressions be shipped to worker processes via :func:`pickle.dumps` / :func:`pickle.loads`. Built-in functions and Python UDFs (scalar, aggregate, window) travel inside the pickle bytes; only FFI-capsule UDFs require pre-registration on the worker. The worker's :class:`SessionContext` for resolving those references is looked up via :func:`datafusion.ipc.set_worker_ctx`, falling back to the global :class:`SessionContext` if none has been installed on the worker. .. warning:: Security :func:`pickle.loads` on the returned tuple executes arbitrary Python on the receiver, including any cloudpickled UDF callable embedded in the payload. Only unpickle expressions from trusted sources. .. warning:: Portability Sender and receiver must run the same Python ``(major, minor)`` version; cloudpickle bytecode is not portable across minor versions. See :meth:`to_bytes` for details on what travels by value vs. by reference. Examples: >>> import pickle >>> from datafusion import col, lit >>> e = col("a") * lit(2) >>> pickle.loads(pickle.dumps(e)).canonical_name() 'a * Int64(2)' The encoding side honors a driver-side sender context installed via :func:`datafusion.ipc.set_sender_ctx` — that is how :meth:`SessionContext.with_python_udf_inlining` propagates through ``pickle.dumps``. The sender context is read by ``__reduce__``, so :func:`copy.copy` and :func:`copy.deepcopy` — which also go through ``__reduce__`` — pick it up too. """ return (Expr._reconstruct, (self.to_bytes(get_sender_ctx()),)) @classmethod def _reconstruct(cls, proto_bytes: bytes) -> Expr: """Internal entry point used by :meth:`__reduce__` on unpickle. Examples: >>> from datafusion import Expr, col, lit >>> blob = (col("a") + lit(1)).to_bytes() >>> Expr._reconstruct(blob).canonical_name() 'a + Int64(1)' """ return cls.from_bytes(proto_bytes) def __richcmp__(self, other: Expr, op: int) -> Expr: """Comparison operator.""" return Expr(self.expr.__richcmp__(other.expr, op)) def __repr__(self) -> str: """Generate a string representation of this expression.""" return self.expr.__repr__() def __add__(self, rhs: Any) -> Expr: """Addition operator. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__add__(rhs.expr)) def __sub__(self, rhs: Any) -> Expr: """Subtraction operator. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__sub__(rhs.expr)) def __truediv__(self, rhs: Any) -> Expr: """Division operator. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__truediv__(rhs.expr)) def __mul__(self, rhs: Any) -> Expr: """Multiplication operator. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__mul__(rhs.expr)) def __mod__(self, rhs: Any) -> Expr: """Modulo operator (%). Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__mod__(rhs.expr)) def __and__(self, rhs: Expr) -> Expr: """Logical AND.""" if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__and__(rhs.expr)) def __or__(self, rhs: Expr) -> Expr: """Logical OR.""" if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__or__(rhs.expr)) def __invert__(self) -> Expr: """Binary not (~).""" return Expr(self.expr.__invert__()) def __getitem__(self, key: str | int) -> Expr: """Retrieve sub-object. If ``key`` is a string, returns the subfield of the struct. If ``key`` is an integer, retrieves the element in the array. Note that the element index begins at ``0``, unlike :py:func:`~datafusion.functions.array_element` which begins at ``1``. If ``key`` is a slice, returns an array that contains a slice of the original array. Similar to integer indexing, this follows Python convention where the index begins at ``0`` unlike :py:func:`~datafusion.functions.array_slice` which begins at ``1``. """ if isinstance(key, int): return Expr( functions_internal.array_element(self.expr, Expr.literal(key + 1).expr) ) if isinstance(key, slice): if isinstance(key.start, int): start = Expr.literal(key.start + 1).expr elif isinstance(key.start, Expr): start = (key.start + Expr.literal(1)).expr else: # Default start at the first element, index 1 start = Expr.literal(1).expr if isinstance(key.stop, int): stop = Expr.literal(key.stop).expr else: stop = key.stop.expr if isinstance(key.step, int): step = Expr.literal(key.step).expr elif isinstance(key.step, Expr): step = key.step.expr else: step = key.step return Expr(functions_internal.array_slice(self.expr, start, stop, step)) return Expr(self.expr.__getitem__(key)) def __eq__(self, rhs: object) -> Expr: """Equal to. Accepts either an expression or any valid PyArrow scalar literal value. """ if rhs is None: return self.is_null() if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__eq__(rhs.expr)) def __ne__(self, rhs: object) -> Expr: """Not equal to. Accepts either an expression or any valid PyArrow scalar literal value. """ if rhs is None: return self.is_not_null() if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__ne__(rhs.expr)) def __ge__(self, rhs: Any) -> Expr: """Greater than or equal to. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__ge__(rhs.expr)) def __gt__(self, rhs: Any) -> Expr: """Greater than. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__gt__(rhs.expr)) def __le__(self, rhs: Any) -> Expr: """Less than or equal to. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__le__(rhs.expr)) def __lt__(self, rhs: Any) -> Expr: """Less than. Accepts either an expression or any valid PyArrow scalar literal value. """ if not isinstance(rhs, Expr): rhs = Expr.literal(rhs) return Expr(self.expr.__lt__(rhs.expr)) __radd__ = __add__ __rand__ = __and__ __rmod__ = __mod__ __rmul__ = __mul__ __ror__ = __or__ __rsub__ = __sub__ __rtruediv__ = __truediv__ @staticmethod def literal(value: Any) -> Expr: """Creates a new expression representing a scalar value. ``value`` must be a valid PyArrow scalar value or easily castable to one. """ if isinstance(value, str): value = pa.scalar(value, type=pa.string_view()) return Expr(expr_internal.RawExpr.literal(value)) @staticmethod def literal_with_metadata(value: Any, metadata: dict[str, str]) -> Expr: """Creates a new expression representing a scalar value with metadata. Args: value: A valid PyArrow scalar value or easily castable to one. metadata: Metadata to attach to the expression. """ if isinstance(value, str): value = pa.scalar(value, type=pa.string_view()) return Expr(expr_internal.RawExpr.literal_with_metadata(value, metadata)) @staticmethod def string_literal(value: str) -> Expr: """Creates a new expression representing a UTF8 literal value. It is different from `literal` because it is pa.string() instead of pa.string_view() This is needed for cases where DataFusion is expecting a UTF8 instead of UTF8View literal, like in: https://github.com/apache/datafusion/blob/86740bfd3d9831d6b7c1d0e1bf4a21d91598a0ac/datafusion/functions/src/core/arrow_cast.rs#L179 """ if isinstance(value, str): value = pa.scalar(value, type=pa.string()) return Expr(expr_internal.RawExpr.literal(value)) return Expr.literal(value) @staticmethod def column(value: str) -> Expr: """Creates a new expression representing a column.""" return Expr(expr_internal.RawExpr.column(value)) def alias(self, name: str, metadata: dict[str, str] | None = None) -> Expr: """Assign a name to the expression. Args: name: The name to assign to the expression. metadata: Optional metadata to attach to the expression. Returns: A new expression with the assigned name. """ return Expr(self.expr.alias(name, metadata)) def sort(self, ascending: bool = True, nulls_first: bool = True) -> SortExpr: """Creates a sort :py:class:`Expr` from an existing :py:class:`Expr`. Args: ascending: If true, sort in ascending order. nulls_first: Return null values first. """ return SortExpr(self, ascending=ascending, nulls_first=nulls_first) def is_null(self) -> Expr: """Returns ``True`` if this expression is null.""" return Expr(self.expr.is_null()) def is_not_null(self) -> Expr: """Returns ``True`` if this expression is not null.""" return Expr(self.expr.is_not_null()) def fill_nan(self, value: Any | Expr | None = None) -> Expr: """Fill NaN values with a provided value.""" if not isinstance(value, Expr): value = Expr.literal(value) return Expr(functions_internal.nanvl(self.expr, value.expr)) def fill_null(self, value: Any | Expr | None = None) -> Expr: """Fill NULL values with a provided value.""" if not isinstance(value, Expr): value = Expr.literal(value) return Expr(functions_internal.nvl(self.expr, value.expr)) _to_pyarrow_types: ClassVar[dict[type, pa.DataType]] = { float: pa.float64(), int: pa.int64(), str: pa.string(), bool: pa.bool_(), } def cast(self, to: pa.DataType[Any] | type) -> Expr: """Cast to a new data type.""" if not isinstance(to, pa.DataType): try: to = self._to_pyarrow_types[to] except KeyError as err: error_msg = "Expected instance of pyarrow.DataType or builtins.type" raise TypeError(error_msg) from err return Expr(self.expr.cast(to)) def try_cast(self, to: pa.DataType[Any] | type) -> Expr: """Cast to a new data type, returning NULL on failure. Like :py:meth:`cast` but produces NULL instead of erroring when the cast cannot be performed for a given row. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["oops"]}) >>> result = df.select(col("a").try_cast(pa.float64()).alias("c")) >>> result.collect_column("c")[0].as_py() is None True """ if not isinstance(to, pa.DataType): try: to = self._to_pyarrow_types[to] except KeyError as err: error_msg = "Expected instance of pyarrow.DataType or builtins.type" raise TypeError(error_msg) from err return Expr(self.expr.try_cast(to)) def between(self, low: Any, high: Any, negated: bool = False) -> Expr: """Returns ``True`` if this expression is between a given range. Args: low: lower bound of the range (inclusive). high: higher bound of the range (inclusive). negated: negates whether the expression is between a given range """ if not isinstance(low, Expr): low = Expr.literal(low) if not isinstance(high, Expr): high = Expr.literal(high) return Expr(self.expr.between(low.expr, high.expr, negated=negated)) def rex_type(self) -> RexType: """Return the Rex Type of this expression. A Rex (Row Expression) specifies a single row of data.That specification could include user defined functions or types. RexType identifies the row as one of the possible valid ``RexType``. """ return self.expr.rex_type() def types(self) -> DataTypeMap: """Return the ``DataTypeMap``. Returns: DataTypeMap which represents the PythonType, Arrow DataType, and SqlType Enum which this expression represents. """ return self.expr.types() def python_value(self) -> Any: """Extracts the Expr value into `Any`. This is only valid for literal expressions. Returns: Python object representing literal value of the expression. """ return self.expr.python_value() def rex_call_operands(self) -> list[Expr]: """Return the operands of the expression based on it's variant type. Row expressions, Rex(s), operate on the concept of operands. Different variants of Expressions, Expr(s), store those operands in different datastructures. This function examines the Expr variant and returns the operands to the calling logic. """ return [Expr(e) for e in self.expr.rex_call_operands()] def rex_call_operator(self) -> str: """Extracts the operator associated with a row expression type call.""" return self.expr.rex_call_operator() def column_name(self, plan: LogicalPlan) -> str: """Compute the output column name based on the provided logical plan.""" return self.expr.column_name(plan._raw_plan) def order_by(self, *exprs: Expr | SortExpr) -> ExprFuncBuilder: """Set the ordering for a window or aggregate function. This function will create an :py:class:`ExprFuncBuilder` that can be used to set parameters for either window or aggregate functions. If used on any other type of expression, an error will be generated when ``build()`` is called. """ return ExprFuncBuilder(self.expr.order_by([sort_or_default(e) for e in exprs])) def filter(self, filter: Expr) -> ExprFuncBuilder: """Filter an aggregate function. This function will create an :py:class:`ExprFuncBuilder` that can be used to set parameters for either window or aggregate functions. If used on any other type of expression, an error will be generated when ``build()`` is called. """ return ExprFuncBuilder(self.expr.filter(filter.expr)) def distinct(self) -> ExprFuncBuilder: """Only evaluate distinct values for an aggregate function. This function will create an :py:class:`ExprFuncBuilder` that can be used to set parameters for either window or aggregate functions. If used on any other type of expression, an error will be generated when ``build()`` is called. """ return ExprFuncBuilder(self.expr.distinct()) def null_treatment(self, null_treatment: NullTreatment) -> ExprFuncBuilder: """Set the treatment for ``null`` values for a window or aggregate function. This function will create an :py:class:`ExprFuncBuilder` that can be used to set parameters for either window or aggregate functions. If used on any other type of expression, an error will be generated when ``build()`` is called. """ return ExprFuncBuilder(self.expr.null_treatment(null_treatment.value)) def partition_by(self, *partition_by: Expr) -> ExprFuncBuilder: """Set the partitioning for a window function. This function will create an :py:class:`ExprFuncBuilder` that can be used to set parameters for either window or aggregate functions. If used on any other type of expression, an error will be generated when ``build()`` is called. """ return ExprFuncBuilder(self.expr.partition_by([e.expr for e in partition_by])) def window_frame(self, window_frame: WindowFrame) -> ExprFuncBuilder: """Set the frame fora window function. This function will create an :py:class:`ExprFuncBuilder` that can be used to set parameters for either window or aggregate functions. If used on any other type of expression, an error will be generated when ``build()`` is called. """ return ExprFuncBuilder(self.expr.window_frame(window_frame.window_frame)) def over(self, window: Window) -> Expr: """Turn an aggregate function into a window function. This function turns any aggregate function into a window function. With the exception of ``partition_by``, how each of the parameters is used is determined by the underlying aggregate function. Args: window: Window definition """ partition_by_raw = expr_list_to_raw_expr_list(window._partition_by) order_by_raw = window._order_by window_frame_raw = ( window._window_frame.window_frame if window._window_frame is not None else None ) null_treatment_raw = ( window._null_treatment.value if window._null_treatment is not None else None ) return Expr( self.expr.over( partition_by=partition_by_raw, order_by=order_by_raw, window_frame=window_frame_raw, null_treatment=null_treatment_raw, ) ) def asin(self) -> Expr: """Returns the arc sine or inverse sine of a number.""" from . import functions as F return F.asin(self) def array_pop_back(self) -> Expr: """Returns the array without the last element.""" from . import functions as F return F.array_pop_back(self) def reverse(self) -> Expr: """Reverse the string argument.""" from . import functions as F return F.reverse(self) def bit_length(self) -> Expr: """Returns the number of bits in the string argument.""" from . import functions as F return F.bit_length(self) def array_length(self) -> Expr: """Returns the length of the array.""" from . import functions as F return F.array_length(self) def array_ndims(self) -> Expr: """Returns the number of dimensions of the array.""" from . import functions as F return F.array_ndims(self) def to_hex(self) -> Expr: """Converts an integer to a hexadecimal string.""" from . import functions as F return F.to_hex(self) def array_dims(self) -> Expr: """Returns an array of the array's dimensions.""" from . import functions as F return F.array_dims(self) def from_unixtime(self) -> Expr: """Converts an integer to RFC3339 timestamp format string.""" from . import functions as F return F.from_unixtime(self) def array_empty(self) -> Expr: """Returns a boolean indicating whether the array is empty.""" from . import functions as F return F.array_empty(self) def sin(self) -> Expr: """Returns the sine of the argument.""" from . import functions as F return F.sin(self) def log10(self) -> Expr: """Base 10 logarithm of the argument.""" from . import functions as F return F.log10(self) def initcap(self) -> Expr: """Set the initial letter of each word to capital. Converts the first letter of each word in ``string`` to uppercase and the remaining characters to lowercase. """ from . import functions as F return F.initcap(self) def list_distinct(self) -> Expr: """Returns distinct values from the array after removing duplicates. This is an alias for :py:func:`array_distinct`. """ from . import functions as F return F.list_distinct(self) def iszero(self) -> Expr: """Returns true if a given number is +0.0 or -0.0 otherwise returns false.""" from . import functions as F return F.iszero(self) def array_distinct(self) -> Expr: """Returns distinct values from the array after removing duplicates.""" from . import functions as F return F.array_distinct(self) def arrow_typeof(self) -> Expr: """Returns the Arrow type of the expression.""" from . import functions as F return F.arrow_typeof(self) def length(self) -> Expr: """The number of characters in the ``string``.""" from . import functions as F return F.length(self) def lower(self) -> Expr: """Converts a string to lowercase.""" from . import functions as F return F.lower(self) def acos(self) -> Expr: """Returns the arc cosine or inverse cosine of a number. Returns: -------- Expr A new expression representing the arc cosine of the input expression. """ from . import functions as F return F.acos(self) def ascii(self) -> Expr: """Returns the numeric code of the first character of the argument.""" from . import functions as F return F.ascii(self) def sha384(self) -> Expr: """Computes the SHA-384 hash of a binary string.""" from . import functions as F return F.sha384(self) def isnan(self) -> Expr: """Returns true if a given number is +NaN or -NaN otherwise returns false.""" from . import functions as F return F.isnan(self) def degrees(self) -> Expr: """Converts the argument from radians to degrees.""" from . import functions as F return F.degrees(self) def cardinality(self) -> Expr: """Returns the total number of elements in the array.""" from . import functions as F return F.cardinality(self) def sha224(self) -> Expr: """Computes the SHA-224 hash of a binary string.""" from . import functions as F return F.sha224(self) def asinh(self) -> Expr: """Returns inverse hyperbolic sine.""" from . import functions as F return F.asinh(self) def flatten(self) -> Expr: """Flattens an array of arrays into a single array.""" from . import functions as F return F.flatten(self) def exp(self) -> Expr: """Returns the exponential of the argument.""" from . import functions as F return F.exp(self) def abs(self) -> Expr: """Return the absolute value of a given number. Returns: -------- Expr A new expression representing the absolute value of the input expression. """ from . import functions as F return F.abs(self) def btrim(self) -> Expr: """Removes all characters, spaces by default, from both sides of a string.""" from . import functions as F return F.btrim(self) def md5(self) -> Expr: """Computes an MD5 128-bit checksum for a string expression.""" from . import functions as F return F.md5(self) def octet_length(self) -> Expr: """Returns the number of bytes of a string.""" from . import functions as F return F.octet_length(self) def cosh(self) -> Expr: """Returns the hyperbolic cosine of the argument.""" from . import functions as F return F.cosh(self) def radians(self) -> Expr: """Converts the argument from degrees to radians.""" from . import functions as F return F.radians(self) def sqrt(self) -> Expr: """Returns the square root of the argument.""" from . import functions as F return F.sqrt(self) def character_length(self) -> Expr: """Returns the number of characters in the argument.""" from . import functions as F return F.character_length(self) def tanh(self) -> Expr: """Returns the hyperbolic tangent of the argument.""" from . import functions as F return F.tanh(self) def atan(self) -> Expr: """Returns inverse tangent of a number.""" from . import functions as F return F.atan(self) def rtrim(self) -> Expr: """Removes all characters, spaces by default, from the end of a string.""" from . import functions as F return F.rtrim(self) def atanh(self) -> Expr: """Returns inverse hyperbolic tangent.""" from . import functions as F return F.atanh(self) def list_dims(self) -> Expr: """Returns an array of the array's dimensions. This is an alias for :py:func:`array_dims`. """ from . import functions as F return F.list_dims(self) def sha256(self) -> Expr: """Computes the SHA-256 hash of a binary string.""" from . import functions as F return F.sha256(self) def factorial(self) -> Expr: """Returns the factorial of the argument.""" from . import functions as F return F.factorial(self) def acosh(self) -> Expr: """Returns inverse hyperbolic cosine.""" from . import functions as F return F.acosh(self) def floor(self) -> Expr: """Returns the nearest integer less than or equal to the argument.""" from . import functions as F return F.floor(self) def ceil(self) -> Expr: """Returns the nearest integer greater than or equal to argument.""" from . import functions as F return F.ceil(self) def list_length(self) -> Expr: """Returns the length of the array. This is an alias for :py:func:`array_length`. """ from . import functions as F return F.list_length(self) def upper(self) -> Expr: """Converts a string to uppercase.""" from . import functions as F return F.upper(self) def chr(self) -> Expr: """Converts the Unicode code point to a UTF8 character.""" from . import functions as F return F.chr(self) def ln(self) -> Expr: """Returns the natural logarithm (base e) of the argument.""" from . import functions as F return F.ln(self) def tan(self) -> Expr: """Returns the tangent of the argument.""" from . import functions as F return F.tan(self) def array_pop_front(self) -> Expr: """Returns the array without the first element.""" from . import functions as F return F.array_pop_front(self) def cbrt(self) -> Expr: """Returns the cube root of a number.""" from . import functions as F return F.cbrt(self) def sha512(self) -> Expr: """Computes the SHA-512 hash of a binary string.""" from . import functions as F return F.sha512(self) def char_length(self) -> Expr: """The number of characters in the ``string``.""" from . import functions as F return F.char_length(self) def list_ndims(self) -> Expr: """Returns the number of dimensions of the array. This is an alias for :py:func:`array_ndims`. """ from . import functions as F return F.list_ndims(self) def trim(self) -> Expr: """Removes all characters, spaces by default, from both sides of a string.""" from . import functions as F return F.trim(self) def cos(self) -> Expr: """Returns the cosine of the argument.""" from . import functions as F return F.cos(self) def sinh(self) -> Expr: """Returns the hyperbolic sine of the argument.""" from . import functions as F return F.sinh(self) def empty(self) -> Expr: """This is an alias for :py:func:`array_empty`.""" from . import functions as F return F.empty(self) def ltrim(self) -> Expr: """Removes all characters, spaces by default, from the beginning of a string.""" from . import functions as F return F.ltrim(self) def signum(self) -> Expr: """Returns the sign of the argument (-1, 0, +1).""" from . import functions as F return F.signum(self) def log2(self) -> Expr: """Base 2 logarithm of the argument.""" from . import functions as F return F.log2(self) def cot(self) -> Expr: """Returns the cotangent of the argument.""" from . import functions as F return F.cot(self) class ExprFuncBuilder: def __init__(self, builder: expr_internal.ExprFuncBuilder) -> None: self.builder = builder def order_by(self, *exprs: Expr) -> ExprFuncBuilder: """Set the ordering for a window or aggregate function. Values given in ``exprs`` must be sort expressions. You can convert any other expression to a sort expression using `.sort()`. """ return ExprFuncBuilder( self.builder.order_by([sort_or_default(e) for e in exprs]) ) def filter(self, filter: Expr) -> ExprFuncBuilder: """Filter values during aggregation.""" return ExprFuncBuilder(self.builder.filter(filter.expr)) def distinct(self) -> ExprFuncBuilder: """Only evaluate distinct values during aggregation.""" return ExprFuncBuilder(self.builder.distinct()) def null_treatment(self, null_treatment: NullTreatment) -> ExprFuncBuilder: """Set how nulls are treated for either window or aggregate functions.""" return ExprFuncBuilder(self.builder.null_treatment(null_treatment.value)) def partition_by(self, *partition_by: Expr) -> ExprFuncBuilder: """Set partitioning for window functions.""" return ExprFuncBuilder( self.builder.partition_by([e.expr for e in partition_by]) ) def window_frame(self, window_frame: WindowFrame) -> ExprFuncBuilder: """Set window frame for window functions.""" return ExprFuncBuilder(self.builder.window_frame(window_frame.window_frame)) def build(self) -> Expr: """Create an expression from a Function Builder.""" return Expr(self.builder.build()) class Window: """Define reusable window parameters.""" def __init__( self, partition_by: list[Expr] | Expr | None = None, window_frame: WindowFrame | None = None, order_by: list[SortExpr | Expr | str] | Expr | SortExpr | str | None = None, null_treatment: NullTreatment | None = None, ) -> None: """Construct a window definition. Args: partition_by: Partitions for window operation window_frame: Define the start and end bounds of the window frame order_by: Set ordering null_treatment: Indicate how nulls are to be treated """ self._partition_by = partition_by self._window_frame = window_frame self._order_by = sort_list_to_raw_sort_list(order_by) self._null_treatment = null_treatment class WindowFrame: """Defines a window frame for performing window operations.""" def __init__( self, units: str, start_bound: Any | None, end_bound: Any | None ) -> None: """Construct a window frame using the given parameters. Args: units: Should be one of ``rows``, ``range``, or ``groups``. start_bound: Sets the preceding bound. Must be >= 0. If none, this will be set to unbounded. If unit type is ``groups``, this parameter must be set. end_bound: Sets the following bound. Must be >= 0. If none, this will be set to unbounded. If unit type is ``groups``, this parameter must be set. """ if not isinstance(start_bound, pa.Scalar) and start_bound is not None: start_bound = pa.scalar(start_bound) if units in ("rows", "groups"): start_bound = start_bound.cast(pa.uint64()) if not isinstance(end_bound, pa.Scalar) and end_bound is not None: end_bound = pa.scalar(end_bound) if units in ("rows", "groups"): end_bound = end_bound.cast(pa.uint64()) self.window_frame = expr_internal.WindowFrame(units, start_bound, end_bound) def __repr__(self) -> str: """Print a string representation of the window frame.""" return self.window_frame.__repr__() def get_frame_units(self) -> str: """Returns the window frame units for the bounds.""" return self.window_frame.get_frame_units() def get_lower_bound(self) -> WindowFrameBound: """Returns starting bound.""" return WindowFrameBound(self.window_frame.get_lower_bound()) def get_upper_bound(self) -> WindowFrameBound: """Returns end bound.""" return WindowFrameBound(self.window_frame.get_upper_bound()) class WindowFrameBound: """Defines a single window frame bound. :py:class:`WindowFrame` typically requires a start and end bound. """ def __init__(self, frame_bound: expr_internal.WindowFrameBound) -> None: """Constructs a window frame bound.""" self.frame_bound = frame_bound def get_offset(self) -> int | None: """Returns the offset of the window frame.""" return self.frame_bound.get_offset() def is_current_row(self) -> bool: """Returns if the frame bound is current row.""" return self.frame_bound.is_current_row() def is_following(self) -> bool: """Returns if the frame bound is following.""" return self.frame_bound.is_following() def is_preceding(self) -> bool: """Returns if the frame bound is preceding.""" return self.frame_bound.is_preceding() def is_unbounded(self) -> bool: """Returns if the frame bound is unbounded.""" return self.frame_bound.is_unbounded() class CaseBuilder: """Builder class for constructing case statements. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.select( ... dfn.functions.case(dfn.col("a")) ... .when(dfn.lit(1), dfn.lit("One")) ... .when(dfn.lit(2), dfn.lit("Two")) ... .otherwise(dfn.lit("Other")) ... .alias("label") ... ) >>> result.to_pydict() {'label': ['One', 'Two', 'Other']} """ def __init__(self, case_builder: expr_internal.CaseBuilder) -> None: """Constructs a case builder. This is not typically called by the end user directly. See :py:func:`datafusion.functions.case` instead. """ self.case_builder = case_builder def when(self, when_expr: Expr, then_expr: Expr) -> CaseBuilder: """Add a case to match against.""" return CaseBuilder(self.case_builder.when(when_expr.expr, then_expr.expr)) def otherwise(self, else_expr: Expr) -> Expr: """Set a default value for the case statement.""" return Expr(self.case_builder.otherwise(else_expr.expr)) def end(self) -> Expr: """Finish building a case statement. Any non-matching cases will end in a `null` value. """ return Expr(self.case_builder.end()) class SortExpr: """Used to specify sorting on either a DataFrame or function.""" def __init__(self, expr: Expr, ascending: bool, nulls_first: bool) -> None: """This constructor should not be called by the end user.""" self.raw_sort = expr_internal.SortExpr(expr.expr, ascending, nulls_first) def expr(self) -> Expr: """Return the raw expr backing the SortExpr.""" return Expr(self.raw_sort.expr()) def ascending(self) -> bool: """Return ascending property.""" return self.raw_sort.ascending() def nulls_first(self) -> bool: """Return nulls_first property.""" return self.raw_sort.nulls_first() def __repr__(self) -> str: """Generate a string representation of this expression.""" return self.raw_sort.__repr__() SortKey = Expr | SortExpr | str class GroupingSet: """Factory for creating grouping set expressions. Grouping sets control how :py:meth:`~datafusion.dataframe.DataFrame.aggregate` groups rows. Instead of a single ``GROUP BY``, they produce multiple grouping levels in one pass — subtotals, cross-tabulations, or arbitrary column subsets. Use :py:func:`~datafusion.functions.grouping` in the aggregate list to tell which columns are aggregated across in each result row. """ @staticmethod def rollup(*exprs: Expr | str) -> Expr: """Create a ``ROLLUP`` grouping set for use with ``aggregate()``. ``ROLLUP`` generates all prefixes of the given column list as grouping sets. For example, ``rollup(a, b)`` produces grouping sets ``(a, b)``, ``(a)``, and ``()`` (grand total). This is equivalent to ``GROUP BY ROLLUP(a, b)`` in SQL. Args: *exprs: Column expressions or column name strings to include in the rollup. Examples: >>> from datafusion.expr import GroupingSet >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 1, 2], "b": [10, 20, 30]}) >>> result = df.aggregate( ... [GroupingSet.rollup(dfn.col("a"))], ... [dfn.functions.sum(dfn.col("b")).alias("s"), ... dfn.functions.grouping(dfn.col("a"))], ... ).sort(dfn.col("a").sort(nulls_first=False)) >>> result.collect_column("s").to_pylist() [30, 30, 60] See Also: :py:meth:`cube`, :py:meth:`grouping_sets`, :py:func:`~datafusion.functions.grouping` """ args = [_to_raw_expr(e) for e in exprs] return Expr(expr_internal.GroupingSet.rollup(*args)) @staticmethod def cube(*exprs: Expr | str) -> Expr: """Create a ``CUBE`` grouping set for use with ``aggregate()``. ``CUBE`` generates all possible subsets of the given column list as grouping sets. For example, ``cube(a, b)`` produces grouping sets ``(a, b)``, ``(a)``, ``(b)``, and ``()`` (grand total). This is equivalent to ``GROUP BY CUBE(a, b)`` in SQL. Args: *exprs: Column expressions or column name strings to include in the cube. Examples: With a single column, ``cube`` behaves identically to :py:meth:`rollup`: >>> from datafusion.expr import GroupingSet >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 1, 2], "b": [10, 20, 30]}) >>> result = df.aggregate( ... [GroupingSet.cube(dfn.col("a"))], ... [dfn.functions.sum(dfn.col("b")).alias("s"), ... dfn.functions.grouping(dfn.col("a"))], ... ).sort(dfn.col("a").sort(nulls_first=False)) >>> result.collect_column("s").to_pylist() [30, 30, 60] See Also: :py:meth:`rollup`, :py:meth:`grouping_sets`, :py:func:`~datafusion.functions.grouping` """ args = [_to_raw_expr(e) for e in exprs] return Expr(expr_internal.GroupingSet.cube(*args)) @staticmethod def grouping_sets(*expr_lists: list[Expr | str]) -> Expr: """Create explicit grouping sets for use with ``aggregate()``. Each argument is a list of column expressions or column name strings representing one grouping set. For example, ``grouping_sets([a], [b])`` groups by ``a`` alone and by ``b`` alone in a single query. This is equivalent to ``GROUP BY GROUPING SETS ((a), (b))`` in SQL. Args: *expr_lists: Each positional argument is a list of expressions or column name strings forming one grouping set. Examples: >>> from datafusion.expr import GroupingSet >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict( ... {"a": ["x", "x", "y"], "b": ["m", "n", "m"], ... "c": [1, 2, 3]}) >>> result = df.aggregate( ... [GroupingSet.grouping_sets( ... [dfn.col("a")], [dfn.col("b")])], ... [dfn.functions.sum(dfn.col("c")).alias("s"), ... dfn.functions.grouping(dfn.col("a")), ... dfn.functions.grouping(dfn.col("b"))], ... ).sort( ... dfn.col("a").sort(nulls_first=False), ... dfn.col("b").sort(nulls_first=False), ... ) >>> result.collect_column("s").to_pylist() [3, 3, 4, 2] See Also: :py:meth:`rollup`, :py:meth:`cube`, :py:func:`~datafusion.functions.grouping` """ raw_lists = [[_to_raw_expr(e) for e in lst] for lst in expr_lists] return Expr(expr_internal.GroupingSet.grouping_sets(*raw_lists)) apache-datafusion-python-54.0.0/python/datafusion/functions/000077500000000000000000000000001521727353300241715ustar00rootroot00000000000000apache-datafusion-python-54.0.0/python/datafusion/functions/__init__.py000066400000000000000000006772601521727353300263240ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Scalar, aggregate, and window functions for :py:class:`~datafusion.expr.Expr`. Each function returns an :py:class:`~datafusion.expr.Expr` that can be combined with other expressions and passed to :py:class:`~datafusion.dataframe.DataFrame` methods such as :py:meth:`~datafusion.dataframe.DataFrame.select`, :py:meth:`~datafusion.dataframe.DataFrame.filter`, :py:meth:`~datafusion.dataframe.DataFrame.aggregate`, and :py:meth:`~datafusion.dataframe.DataFrame.window`. The module is conventionally imported as ``F`` so calls read like ``F.sum(col("price"))``. Examples: >>> from datafusion import functions as F >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3, 4]}) >>> df.aggregate([], [F.sum(col("a")).alias("total")]).to_pydict() {'total': [10]} See :ref:`aggregation` and :ref:`window_functions` in the online documentation for categorized catalogs of aggregate and window functions. """ from __future__ import annotations import inspect import warnings from typing import TYPE_CHECKING, Any import pyarrow as pa if TYPE_CHECKING: from collections.abc import Callable, Iterable from datafusion._internal import functions as f from datafusion.common import NullTreatment from datafusion.expr import ( CaseBuilder, Expr, SortExpr, SortKey, coerce_to_expr, coerce_to_expr_list, coerce_to_expr_or_none, expr_list_to_raw_expr_list, sort_list_to_raw_sort_list, sort_or_default, ) from datafusion.functions import spark def _warn_if_expr_for_literal_arg( value: Any, function_name: str, arg_name: str ) -> None: if isinstance(value, Expr): warnings.warn( f"Passing Expr for {function_name}() argument {arg_name!r} is deprecated; " "pass a Python literal instead.", DeprecationWarning, stacklevel=3, ) __all__ = [ "abs", "acos", "acosh", "alias", "any_match", "approx_distinct", "approx_median", "approx_percentile_cont", "approx_percentile_cont_with_weight", "array", "array_agg", "array_any_match", "array_any_value", "array_append", "array_cat", "array_compact", "array_concat", "array_contains", "array_dims", "array_distance", "array_distinct", "array_element", "array_empty", "array_except", "array_extract", "array_filter", "array_has", "array_has_all", "array_has_any", "array_indexof", "array_intersect", "array_join", "array_length", "array_max", "array_min", "array_ndims", "array_normalize", "array_pop_back", "array_pop_front", "array_position", "array_positions", "array_prepend", "array_push_back", "array_push_front", "array_remove", "array_remove_all", "array_remove_n", "array_repeat", "array_replace", "array_replace_all", "array_replace_n", "array_resize", "array_reverse", "array_slice", "array_sort", "array_to_string", "array_transform", "array_union", "arrays_overlap", "arrays_zip", "arrow_cast", "arrow_field", "arrow_metadata", "arrow_try_cast", "arrow_typeof", "ascii", "asin", "asinh", "atan", "atan2", "atanh", "avg", "bit_and", "bit_length", "bit_or", "bit_xor", "bool_and", "bool_or", "btrim", "cardinality", "case", "cast_to_type", "cbrt", "ceil", "char_length", "character_length", "chr", "coalesce", "col", "concat", "concat_ws", "contains", "corr", "cos", "cosh", "cosine_distance", "cot", "count", "count_star", "covar", "covar_pop", "covar_samp", "cume_dist", "current_date", "current_time", "current_timestamp", "date_bin", "date_format", "date_part", "date_trunc", "datepart", "datetrunc", "decode", "degrees", "dense_rank", "digest", "dot_product", "element_at", "empty", "encode", "ends_with", "exp", "extract", "factorial", "find_in_set", "first_value", "flatten", "floor", "from_unixtime", "gcd", "gen_series", "generate_series", "get_field", "greatest", "grouping", "ifnull", "in_list", "initcap", "inner_product", "instr", "isnan", "iszero", "lag", "lambda_", "lambda_var", "last_value", "lcm", "lead", "least", "left", "length", "levenshtein", "list_any_match", "list_any_value", "list_append", "list_cat", "list_compact", "list_concat", "list_contains", "list_dims", "list_distance", "list_distinct", "list_element", "list_empty", "list_except", "list_extract", "list_filter", "list_has", "list_has_all", "list_has_any", "list_indexof", "list_intersect", "list_join", "list_length", "list_max", "list_min", "list_ndims", "list_normalize", "list_overlap", "list_pop_back", "list_pop_front", "list_position", "list_positions", "list_prepend", "list_push_back", "list_push_front", "list_remove", "list_remove_all", "list_remove_n", "list_repeat", "list_replace", "list_replace_all", "list_replace_n", "list_resize", "list_reverse", "list_slice", "list_sort", "list_to_string", "list_transform", "list_union", "list_zip", "ln", "log", "log2", "log10", "lower", "lpad", "ltrim", "make_array", "make_date", "make_list", "make_map", "make_time", "map_entries", "map_extract", "map_keys", "map_values", "max", "md5", "mean", "median", "min", "named_struct", "nanvl", "now", "nth_value", "ntile", "nullif", "nvl", "nvl2", "octet_length", "order_by", "overlay", "percent_rank", "percentile_cont", "pi", "position", "pow", "power", "quantile_cont", "radians", "random", "range", "rank", "regexp_count", "regexp_instr", "regexp_like", "regexp_match", "regexp_replace", "regr_avgx", "regr_avgy", "regr_count", "regr_intercept", "regr_r2", "regr_slope", "regr_sxx", "regr_sxy", "regr_syy", "repeat", "replace", "reverse", "right", "round", "row", "row_number", "rpad", "rtrim", "sha224", "sha256", "sha384", "sha512", "signum", "sin", "sinh", "spark", "split_part", "sqrt", "starts_with", "stddev", "stddev_pop", "stddev_samp", "string_agg", "string_to_array", "string_to_list", "strpos", "struct", "substr", "substr_index", "substring", "sum", "tan", "tanh", "to_char", "to_date", "to_hex", "to_local_time", "to_time", "to_timestamp", "to_timestamp_micros", "to_timestamp_millis", "to_timestamp_nanos", "to_timestamp_seconds", "to_unixtime", "today", "translate", "trim", "trunc", "try_cast_to_type", "union_extract", "union_tag", "upper", "uuid", "var", "var_pop", "var_population", "var_samp", "var_sample", "version", "when", "with_metadata", ] def isnan(expr: Expr) -> Expr: """Returns true if a given number is +NaN or -NaN otherwise returns false. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, np.nan]}) >>> result = df.select(dfn.functions.isnan(dfn.col("a")).alias("isnan")) >>> result.collect_column("isnan")[1].as_py() True """ return Expr(f.isnan(expr.expr)) def nullif(expr1: Expr, expr2: Expr) -> Expr: """Returns NULL if expr1 equals expr2; otherwise it returns expr1. This can be used to perform the inverse operation of the COALESCE expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2], "b": [1, 3]}) >>> result = df.select( ... dfn.functions.nullif(dfn.col("a"), dfn.col("b")).alias("nullif")) >>> result.collect_column("nullif").to_pylist() [None, 2] """ return Expr(f.nullif(expr1.expr, expr2.expr)) def encode(expr: Expr, encoding: Expr | str) -> Expr: """Encode the ``input``, using the ``encoding``. encoding can be base64 or hex. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.encode(dfn.col("a"), "base64").alias("enc")) >>> result.collect_column("enc")[0].as_py() 'aGVsbG8' """ _warn_if_expr_for_literal_arg(encoding, "encode", "encoding") encoding = coerce_to_expr(encoding) return Expr(f.encode(expr.expr, encoding.expr)) def decode(expr: Expr, encoding: Expr | str) -> Expr: """Decode the ``input``, using the ``encoding``. encoding can be base64 or hex. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["aGVsbG8="]}) >>> result = df.select( ... dfn.functions.decode(dfn.col("a"), "base64").alias("dec")) >>> result.collect_column("dec")[0].as_py() b'hello' """ _warn_if_expr_for_literal_arg(encoding, "decode", "encoding") encoding = coerce_to_expr(encoding) return Expr(f.decode(expr.expr, encoding.expr)) def array_to_string(expr: Expr, delimiter: Expr | str) -> Expr: """Converts each element to its text representation. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_to_string(dfn.col("a"), ",").alias("s")) >>> result.collect_column("s")[0].as_py() '1,2,3' """ delimiter = coerce_to_expr(delimiter) return Expr(f.array_to_string(expr.expr, delimiter.expr.cast(pa.string()))) def array_join(expr: Expr, delimiter: Expr | str) -> Expr: """Converts each element to its text representation. See Also: This is an alias for :py:func:`array_to_string`. """ return array_to_string(expr, delimiter) def list_to_string(expr: Expr, delimiter: Expr | str) -> Expr: """Converts each element to its text representation. See Also: This is an alias for :py:func:`array_to_string`. """ return array_to_string(expr, delimiter) def list_join(expr: Expr, delimiter: Expr | str) -> Expr: """Converts each element to its text representation. See Also: This is an alias for :py:func:`array_to_string`. """ return array_to_string(expr, delimiter) def lambda_var(name: str) -> Expr: """Create an unresolved reference to a lambda parameter by ``name``. Use this inside the body passed to :py:func:`lambda_` to refer to one of the lambda's parameters. The owning higher-order function (such as :py:func:`array_transform`) binds the variable to a concrete element type during query planning. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> double_fn = F.lambda_(["v"], F.lambda_var("v") * lit(2)) >>> df.select( ... F.array_transform(col("a"), double_fn).alias("d") ... ).collect_column("d")[0].as_py() [2, 4, 6] See Also: :py:func:`lambda_`, :py:func:`array_transform`, :py:func:`array_any_match`. """ return Expr(f.lambda_var(name)) def lambda_(params: list[str], body: Expr) -> Expr: """Create a lambda expression from parameter names and a body expression. This is the explicit form of building a lambda. Most callers can instead pass a Python callable directly to a higher-order function such as :py:func:`array_transform`, which builds the lambda automatically. Reach for ``lambda_`` when you want explicit control over the parameter names. Args: params: Ordered lambda parameter names. body: Body expression that references the parameters via :py:func:`lambda_var`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> double_fn = F.lambda_(["v"], F.lambda_var("v") * lit(2)) >>> df.select( ... F.array_transform(col("a"), double_fn).alias("d") ... ).collect_column("d")[0].as_py() [2, 4, 6] See Also: :py:func:`lambda_var`, :py:func:`array_transform`, :py:func:`array_any_match`. """ return Expr(f.lambda_(params, body.expr)) def _to_lambda(fn: Expr | Callable[..., Any]) -> Expr: """Coerce ``fn`` to a lambda ``Expr``. Accepts either an ``Expr`` produced by :py:func:`lambda_` (returned unchanged) or a Python callable. A callable is introspected for its parameter names; those names become :py:func:`lambda_var` references passed positionally into the callable, and its return value (coerced to an ``Expr``) becomes the lambda body. """ if isinstance(fn, Expr): return fn if not callable(fn): msg = f"expected an Expr or callable, got {type(fn).__name__}" raise TypeError(msg) params = list(inspect.signature(fn).parameters) if not params: msg = "lambda callable must accept at least one parameter" raise ValueError(msg) body = coerce_to_expr(fn(*[lambda_var(p) for p in params])) return lambda_(params, body) def array_transform(array: Expr, transform: Expr | Callable[..., Any]) -> Expr: """Transform each element of ``array`` with a lambda. ``transform`` may be a Python callable, which is converted to a lambda automatically (its parameter names become the lambda parameters), or an explicit lambda built with :py:func:`lambda_`. Examples: Using a Python callable: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> df.select( ... F.array_transform(col("a"), lambda v: v * 2).alias("d") ... ).collect_column("d")[0].as_py() [2, 4, 6] Using an explicit lambda built with :py:func:`lambda_`: >>> double_fn = F.lambda_(["v"], F.lambda_var("v") * lit(2)) >>> df.select( ... F.array_transform(col("a"), double_fn).alias("d") ... ).collect_column("d")[0].as_py() [2, 4, 6] See Also: :py:func:`array_any_match`, :py:func:`lambda_`. """ return Expr(f.array_transform(array.expr, _to_lambda(transform).expr)) def list_transform(array: Expr, transform: Expr | Callable[..., Any]) -> Expr: """Transform each element of a list with a lambda. See Also: This is an alias for :py:func:`array_transform`. """ return array_transform(array, transform) def array_any_match(array: Expr, predicate: Expr | Callable[..., Any]) -> Expr: """Return ``True`` if any element of ``array`` satisfies ``predicate``. ``predicate`` may be a Python callable, converted to a lambda automatically, or an explicit lambda built with :py:func:`lambda_`. It must return a boolean expression. Examples: Using a Python callable: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> df.select( ... F.array_any_match(col("a"), lambda v: v > 2).alias("m") ... ).collect_column("m")[0].as_py() True Using an explicit lambda built with :py:func:`lambda_`: >>> predicate = F.lambda_(["v"], F.lambda_var("v") > lit(2)) >>> df.select( ... F.array_any_match(col("a"), predicate).alias("m") ... ).collect_column("m")[0].as_py() True See Also: :py:func:`array_transform`, :py:func:`lambda_`. """ return Expr(f.array_any_match(array.expr, _to_lambda(predicate).expr)) def any_match(array: Expr, predicate: Expr | Callable[..., Any]) -> Expr: """Return ``True`` if any element of an array satisfies a predicate. See Also: This is an alias for :py:func:`array_any_match`. """ return array_any_match(array, predicate) def list_any_match(array: Expr, predicate: Expr | Callable[..., Any]) -> Expr: """Return ``True`` if any element of a list satisfies a predicate. See Also: This is an alias for :py:func:`array_any_match`. """ return array_any_match(array, predicate) def array_filter(array: Expr, predicate: Expr | Callable[..., Any]) -> Expr: """Keep the elements of ``array`` for which ``predicate`` is ``True``. ``predicate`` may be a Python callable, converted to a lambda automatically, or an explicit lambda built with :py:func:`lambda_`. It must return a boolean expression. The result is a new array containing only the matching elements. Examples: Using a Python callable: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3, 4, 5]]}) >>> df.select( ... F.array_filter(col("a"), lambda v: v > 2).alias("f") ... ).collect_column("f")[0].as_py() [3, 4, 5] Using an explicit lambda built with :py:func:`lambda_`: >>> predicate = F.lambda_(["v"], F.lambda_var("v") > lit(2)) >>> df.select( ... F.array_filter(col("a"), predicate).alias("f") ... ).collect_column("f")[0].as_py() [3, 4, 5] See Also: :py:func:`array_transform`, :py:func:`array_any_match`, :py:func:`lambda_`. """ return Expr(f.array_filter(array.expr, _to_lambda(predicate).expr)) def list_filter(array: Expr, predicate: Expr | Callable[..., Any]) -> Expr: """Keep the elements of a list for which a predicate is ``True``. See Also: This is an alias for :py:func:`array_filter`. """ return array_filter(array, predicate) def in_list(arg: Expr, values: list[Expr], negated: bool = False) -> Expr: """Returns whether the argument is contained within the list ``values``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.select( ... dfn.functions.in_list( ... dfn.col("a"), [dfn.lit(1), dfn.lit(3)] ... ).alias("in") ... ) >>> result.collect_column("in").to_pylist() [True, False, True] >>> result = df.select( ... dfn.functions.in_list( ... dfn.col("a"), [dfn.lit(1), dfn.lit(3)], ... negated=True, ... ).alias("not_in") ... ) >>> result.collect_column("not_in").to_pylist() [False, True, False] """ values = [v.expr for v in values] return Expr(f.in_list(arg.expr, values, negated)) def digest(value: Expr, method: Expr | str) -> Expr: """Computes the binary hash of an expression using the specified algorithm. Standard algorithms are md5, sha224, sha256, sha384, sha512, blake2s, blake2b, and blake3. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.digest(dfn.col("a"), "md5").alias("d")) >>> len(result.collect_column("d")[0].as_py()) > 0 True """ _warn_if_expr_for_literal_arg(method, "digest", "method") method = coerce_to_expr(method) return Expr(f.digest(value.expr, method.expr)) def contains(string: Expr, search_str: Expr | str) -> Expr: """Returns true if ``search_str`` is found within ``string`` (case-sensitive). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["the quick brown fox"]}) >>> result = df.select( ... dfn.functions.contains(dfn.col("a"), "brown").alias("c")) >>> result.collect_column("c")[0].as_py() True """ search_str = coerce_to_expr(search_str) return Expr(f.contains(string.expr, search_str.expr)) def concat(*args: Expr) -> Expr: """Concatenates the text representations of all the arguments. NULL arguments are ignored. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"], "b": [" world"]}) >>> result = df.select( ... dfn.functions.concat(dfn.col("a"), dfn.col("b")).alias("c") ... ) >>> result.collect_column("c")[0].as_py() 'hello world' """ args = [arg.expr for arg in args] return Expr(f.concat(args)) def concat_ws(separator: str, *args: Expr) -> Expr: """Concatenates the list ``args`` with the separator. ``NULL`` arguments are ignored. ``separator`` should not be ``NULL``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"], "b": ["world"]}) >>> result = df.select( ... dfn.functions.concat_ws("-", dfn.col("a"), dfn.col("b")).alias("c")) >>> result.collect_column("c")[0].as_py() 'hello-world' """ args = [arg.expr for arg in args] return Expr(f.concat_ws(separator, args)) def order_by(expr: Expr, ascending: bool = True, nulls_first: bool = True) -> SortExpr: """Creates a new sort expression. Examples: >>> sort_expr = dfn.functions.order_by( ... dfn.col("a"), ascending=False) >>> sort_expr.ascending() False >>> sort_expr = dfn.functions.order_by( ... dfn.col("a"), ascending=True, nulls_first=False) >>> sort_expr.nulls_first() False """ return SortExpr(expr, ascending=ascending, nulls_first=nulls_first) def alias(expr: Expr, name: str, metadata: dict[str, str] | None = None) -> Expr: """Creates an alias expression with an optional metadata dictionary. Args: expr: The expression to alias name: The alias name metadata: Optional metadata to attach to the column Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2]}) >>> result = df.select( ... dfn.functions.alias( ... dfn.col("a"), "b" ... ) ... ) >>> result.collect_column("b")[0].as_py() 1 >>> result = df.select( ... dfn.functions.alias( ... dfn.col("a"), "b", metadata={"info": "test"} ... ) ... ) >>> result.schema() b: int64 -- field metadata -- info: 'test' """ return Expr(f.alias(expr.expr, name, metadata)) def col(name: str) -> Expr: """Creates a column reference expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> df.select(dfn.functions.col("a")).collect_column("a")[0].as_py() 1 """ return Expr(f.col(name)) def count_star(filter: Expr | None = None) -> Expr: """Create a COUNT(1) aggregate expression. This aggregate function will count all of the rows in the partition. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``distinct``, and ``null_treatment``. Args: filter: If provided, only count rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.count_star( ... ).alias("cnt")]) >>> result.collect_column("cnt")[0].as_py() 3 >>> result = df.aggregate( ... [], [dfn.functions.count_star( ... filter=dfn.col("a") > dfn.lit(1) ... ).alias("cnt")]) >>> result.collect_column("cnt")[0].as_py() 2 """ return count(Expr.literal(1), filter=filter) def case(expr: Expr) -> CaseBuilder: """Create a case expression. Create a :py:class:`~datafusion.expr.CaseBuilder` to match cases for the expression ``expr``. See :py:class:`~datafusion.expr.CaseBuilder` for detailed usage. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.select( ... dfn.functions.case(dfn.col("a")).when(dfn.lit(1), ... dfn.lit("one")).otherwise(dfn.lit("other")).alias("c")) >>> result.collect_column("c")[0].as_py() 'one' """ return CaseBuilder(f.case(expr.expr)) def when(when: Expr, then: Expr) -> CaseBuilder: """Create a case expression that has no base expression. Create a :py:class:`~datafusion.expr.CaseBuilder` to match cases for the expression ``expr``. See :py:class:`~datafusion.expr.CaseBuilder` for detailed usage. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.select( ... dfn.functions.when(dfn.col("a") > dfn.lit(2), ... dfn.lit("big")).otherwise(dfn.lit("small")).alias("c")) >>> result.collect_column("c")[2].as_py() 'big' """ return CaseBuilder(f.when(when.expr, then.expr)) # scalar functions def abs(arg: Expr) -> Expr: """Return the absolute value of a given number. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [-1, 0, 1]}) >>> result = df.select(dfn.functions.abs(dfn.col("a")).alias("abs")) >>> result.collect_column("abs")[0].as_py() 1 """ return Expr(f.abs(arg.expr)) def acos(arg: Expr) -> Expr: """Returns the arc cosine or inverse cosine of a number. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0]}) >>> result = df.select(dfn.functions.acos(dfn.col("a")).alias("acos")) >>> result.collect_column("acos")[0].as_py() 0.0 """ return Expr(f.acos(arg.expr)) def acosh(arg: Expr) -> Expr: """Returns inverse hyperbolic cosine. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0]}) >>> result = df.select(dfn.functions.acosh(dfn.col("a")).alias("acosh")) >>> result.collect_column("acosh")[0].as_py() 0.0 """ return Expr(f.acosh(arg.expr)) def ascii(arg: Expr) -> Expr: """Returns the numeric code of the first character of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["a","b","c"]}) >>> ascii_df = df.select(dfn.functions.ascii(dfn.col("a")).alias("ascii")) >>> ascii_df.collect_column("ascii")[0].as_py() 97 """ return Expr(f.ascii(arg.expr)) def asin(arg: Expr) -> Expr: """Returns the arc sine or inverse sine of a number. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.asin(dfn.col("a")).alias("asin")) >>> result.collect_column("asin")[0].as_py() 0.0 """ return Expr(f.asin(arg.expr)) def asinh(arg: Expr) -> Expr: """Returns inverse hyperbolic sine. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.asinh(dfn.col("a")).alias("asinh")) >>> result.collect_column("asinh")[0].as_py() 0.0 """ return Expr(f.asinh(arg.expr)) def atan(arg: Expr) -> Expr: """Returns inverse tangent of a number. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.atan(dfn.col("a")).alias("atan")) >>> result.collect_column("atan")[0].as_py() 0.0 """ return Expr(f.atan(arg.expr)) def atanh(arg: Expr) -> Expr: """Returns inverse hyperbolic tangent. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.atanh(dfn.col("a")).alias("atanh")) >>> result.collect_column("atanh")[0].as_py() 0.0 """ return Expr(f.atanh(arg.expr)) def atan2(y: Expr, x: Expr) -> Expr: """Returns inverse tangent of a division given in the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [0.0], "x": [1.0]}) >>> result = df.select( ... dfn.functions.atan2(dfn.col("y"), dfn.col("x")).alias("atan2")) >>> result.collect_column("atan2")[0].as_py() 0.0 """ return Expr(f.atan2(y.expr, x.expr)) def bit_length(arg: Expr) -> Expr: """Returns the number of bits in the string argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["a","b","c"]}) >>> bit_df = df.select(dfn.functions.bit_length(dfn.col("a")).alias("bit_len")) >>> bit_df.collect_column("bit_len")[0].as_py() 8 """ return Expr(f.bit_length(arg.expr)) def btrim(arg: Expr) -> Expr: """Removes all characters, spaces by default, from both sides of a string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [" a "]}) >>> trim_df = df.select(dfn.functions.btrim(dfn.col("a")).alias("trimmed")) >>> trim_df.collect_column("trimmed")[0].as_py() 'a' """ return Expr(f.btrim(arg.expr)) def cbrt(arg: Expr) -> Expr: """Returns the cube root of a number. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [27]}) >>> cbrt_df = df.select(dfn.functions.cbrt(dfn.col("a")).alias("cbrt")) >>> cbrt_df.collect_column("cbrt")[0].as_py() 3.0 """ return Expr(f.cbrt(arg.expr)) def ceil(arg: Expr) -> Expr: """Returns the nearest integer greater than or equal to argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.9]}) >>> ceil_df = df.select(dfn.functions.ceil(dfn.col("a")).alias("ceil")) >>> ceil_df.collect_column("ceil")[0].as_py() 2.0 """ return Expr(f.ceil(arg.expr)) def character_length(arg: Expr) -> Expr: """Returns the number of characters in the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["abc","b","c"]}) >>> char_len_df = df.select( ... dfn.functions.character_length(dfn.col("a")).alias("char_len")) >>> char_len_df.collect_column("char_len")[0].as_py() 3 """ return Expr(f.character_length(arg.expr)) def length(string: Expr) -> Expr: """The number of characters in the ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.length(dfn.col("a")).alias("len")) >>> result.collect_column("len")[0].as_py() 5 """ return Expr(f.length(string.expr)) def char_length(string: Expr) -> Expr: """The number of characters in the ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.char_length(dfn.col("a")).alias("len")) >>> result.collect_column("len")[0].as_py() 5 """ return Expr(f.char_length(string.expr)) def chr(arg: Expr) -> Expr: """Converts the Unicode code point to a UTF8 character. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [65]}) >>> result = df.select(dfn.functions.chr(dfn.col("a")).alias("chr")) >>> result.collect_column("chr")[0].as_py() 'A' """ return Expr(f.chr(arg.expr)) def coalesce(*args: Expr) -> Expr: """Returns the value of the first expr in ``args`` which is not NULL. Args: *args: Expressions to evaluate in order. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [None, 1], "b": [2, 3]}) >>> result = df.select( ... dfn.functions.coalesce(dfn.col("a"), dfn.col("b")).alias("c")) >>> result.collect_column("c")[0].as_py() 2 """ args = [arg.expr for arg in args] return Expr(f.coalesce(*args)) def cos(arg: Expr) -> Expr: """Returns the cosine of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0,-1,1]}) >>> cos_df = df.select(dfn.functions.cos(dfn.col("a")).alias("cos")) >>> cos_df.collect_column("cos")[0].as_py() 1.0 """ return Expr(f.cos(arg.expr)) def cosh(arg: Expr) -> Expr: """Returns the hyperbolic cosine of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0,-1,1]}) >>> cosh_df = df.select(dfn.functions.cosh(dfn.col("a")).alias("cosh")) >>> cosh_df.collect_column("cosh")[0].as_py() 1.0 """ return Expr(f.cosh(arg.expr)) def cot(arg: Expr) -> Expr: """Returns the cotangent of the argument. Examples: >>> from math import pi >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [pi / 4]}) >>> result = df.select( ... dfn.functions.cot(dfn.col("a")).alias("cot") ... ) >>> result.collect_column("cot")[0].as_py() 1.0... """ return Expr(f.cot(arg.expr)) def degrees(arg: Expr) -> Expr: """Converts the argument from radians to degrees. Examples: >>> from math import pi >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0,pi,2*pi]}) >>> deg_df = df.select(dfn.functions.degrees(dfn.col("a")).alias("deg")) >>> deg_df.collect_column("deg")[2].as_py() 360.0 """ return Expr(f.degrees(arg.expr)) def ends_with(arg: Expr, suffix: Expr | str) -> Expr: """Returns true if the ``string`` ends with the ``suffix``, false otherwise. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["abc","b","c"]}) >>> ends_with_df = df.select( ... dfn.functions.ends_with(dfn.col("a"), "c").alias("ends_with")) >>> ends_with_df.collect_column("ends_with")[0].as_py() True """ suffix = coerce_to_expr(suffix) return Expr(f.ends_with(arg.expr, suffix.expr)) def exp(arg: Expr) -> Expr: """Returns the exponential of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.exp(dfn.col("a")).alias("exp")) >>> result.collect_column("exp")[0].as_py() 1.0 """ return Expr(f.exp(arg.expr)) def factorial(arg: Expr) -> Expr: """Returns the factorial of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [3]}) >>> result = df.select( ... dfn.functions.factorial(dfn.col("a")).alias("factorial") ... ) >>> result.collect_column("factorial")[0].as_py() 6 """ return Expr(f.factorial(arg.expr)) def find_in_set(string: Expr, string_list: Expr | str) -> Expr: """Find a string in a list of strings. Returns a value in the range of 1 to N if the string is in the string list ``string_list`` consisting of N substrings. The string list is a string composed of substrings separated by ``,`` characters. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["b"]}) >>> result = df.select( ... dfn.functions.find_in_set(dfn.col("a"), "a,b,c").alias("pos")) >>> result.collect_column("pos")[0].as_py() 2 """ string_list = coerce_to_expr(string_list) return Expr(f.find_in_set(string.expr, string_list.expr)) def floor(arg: Expr) -> Expr: """Returns the nearest integer less than or equal to the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.9]}) >>> floor_df = df.select(dfn.functions.floor(dfn.col("a")).alias("floor")) >>> floor_df.collect_column("floor")[0].as_py() 1.0 """ return Expr(f.floor(arg.expr)) def gcd(x: Expr, y: Expr) -> Expr: """Returns the greatest common divisor. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [12], "b": [8]}) >>> result = df.select( ... dfn.functions.gcd(dfn.col("a"), dfn.col("b")).alias("gcd") ... ) >>> result.collect_column("gcd")[0].as_py() 4 """ return Expr(f.gcd(x.expr, y.expr)) def greatest(*args: Expr) -> Expr: """Returns the greatest value from a list of expressions. Returns NULL if all expressions are NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 3], "b": [2, 1]}) >>> result = df.select( ... dfn.functions.greatest(dfn.col("a"), dfn.col("b")).alias("greatest")) >>> result.collect_column("greatest")[0].as_py() 2 >>> result.collect_column("greatest")[1].as_py() 3 """ exprs = [arg.expr for arg in args] return Expr(f.greatest(*exprs)) def ifnull(x: Expr, y: Expr) -> Expr: """Returns ``x`` if ``x`` is not NULL. Otherwise returns ``y``. Args: x: Expression to return when it is not NULL. y: Fallback expression to return when ``x`` is NULL. See Also: This is an alias for :py:func:`nvl`. """ return nvl(x, y) def initcap(string: Expr) -> Expr: """Set the initial letter of each word to capital. Converts the first letter of each word in ``string`` to uppercase and the remaining characters to lowercase. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["the cat"]}) >>> cap_df = df.select(dfn.functions.initcap(dfn.col("a")).alias("cap")) >>> cap_df.collect_column("cap")[0].as_py() 'The Cat' """ return Expr(f.initcap(string.expr)) def instr(string: Expr, substring: Expr | str) -> Expr: """Finds the position from where the ``substring`` matches the ``string``. See Also: This is an alias for :py:func:`strpos`. """ return strpos(string, substring) def iszero(arg: Expr) -> Expr: """Returns true if a given number is +0.0 or -0.0 otherwise returns false. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0, 1.0]}) >>> result = df.select(dfn.functions.iszero(dfn.col("a")).alias("iz")) >>> result.collect_column("iz")[0].as_py() True """ return Expr(f.iszero(arg.expr)) def lcm(x: Expr, y: Expr) -> Expr: """Returns the least common multiple. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [4], "b": [6]}) >>> result = df.select( ... dfn.functions.lcm(dfn.col("a"), dfn.col("b")).alias("lcm") ... ) >>> result.collect_column("lcm")[0].as_py() 12 """ return Expr(f.lcm(x.expr, y.expr)) def least(*args: Expr) -> Expr: """Returns the least value from a list of expressions. Returns NULL if all expressions are NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 3], "b": [2, 1]}) >>> result = df.select( ... dfn.functions.least(dfn.col("a"), dfn.col("b")).alias("least")) >>> result.collect_column("least")[0].as_py() 1 >>> result.collect_column("least")[1].as_py() 1 """ exprs = [arg.expr for arg in args] return Expr(f.least(*exprs)) def left(string: Expr, n: Expr | int) -> Expr: """Returns the first ``n`` characters in the ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["the cat"]}) >>> left_df = df.select( ... dfn.functions.left(dfn.col("a"), 3).alias("left")) >>> left_df.collect_column("left")[0].as_py() 'the' """ n = coerce_to_expr(n) return Expr(f.left(string.expr, n.expr)) def levenshtein(string1: Expr, string2: Expr | str) -> Expr: """Returns the Levenshtein distance between the two given strings. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["kitten"]}) >>> result = df.select( ... dfn.functions.levenshtein(dfn.col("a"), "sitting").alias("d")) >>> result.collect_column("d")[0].as_py() 3 """ string2 = coerce_to_expr(string2) return Expr(f.levenshtein(string1.expr, string2.expr)) def ln(arg: Expr) -> Expr: """Returns the natural logarithm (base e) of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0]}) >>> result = df.select(dfn.functions.ln(dfn.col("a")).alias("ln")) >>> result.collect_column("ln")[0].as_py() 0.0 """ return Expr(f.ln(arg.expr)) def log(base: Expr | int | float, num: Expr) -> Expr: # noqa: PYI041 """Returns the logarithm of a number for a particular ``base``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [100.0]}) >>> result = df.select( ... dfn.functions.log(10.0, dfn.col("a")).alias("log") ... ) >>> result.collect_column("log")[0].as_py() 2.0 """ base = coerce_to_expr(base) return Expr(f.log(base.expr, num.expr)) def log10(arg: Expr) -> Expr: """Base 10 logarithm of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [100.0]}) >>> result = df.select(dfn.functions.log10(dfn.col("a")).alias("log10")) >>> result.collect_column("log10")[0].as_py() 2.0 """ return Expr(f.log10(arg.expr)) def log2(arg: Expr) -> Expr: """Base 2 logarithm of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [8.0]}) >>> result = df.select(dfn.functions.log2(dfn.col("a")).alias("log2")) >>> result.collect_column("log2")[0].as_py() 3.0 """ return Expr(f.log2(arg.expr)) def lower(arg: Expr) -> Expr: """Converts a string to lowercase. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["THE CaT"]}) >>> lower_df = df.select(dfn.functions.lower(dfn.col("a")).alias("lower")) >>> lower_df.collect_column("lower")[0].as_py() 'the cat' """ return Expr(f.lower(arg.expr)) def lpad(string: Expr, count: Expr | int, characters: Expr | str | None = None) -> Expr: """Add left padding to a string. Extends the string to length length by prepending the characters fill (a space by default). If the string is already longer than length then it is truncated (on the right). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["the cat", "a hat"]}) >>> lpad_df = df.select( ... dfn.functions.lpad(dfn.col("a"), 6).alias("lpad")) >>> lpad_df.collect_column("lpad")[0].as_py() 'the ca' >>> lpad_df.collect_column("lpad")[1].as_py() ' a hat' >>> result = df.select( ... dfn.functions.lpad( ... dfn.col("a"), 10, characters="." ... ).alias("lpad")) >>> result.collect_column("lpad")[0].as_py() '...the cat' """ count = coerce_to_expr(count) characters = coerce_to_expr(characters if characters is not None else " ") return Expr(f.lpad(string.expr, count.expr, characters.expr)) def ltrim(arg: Expr) -> Expr: """Removes all characters, spaces by default, from the beginning of a string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [" a "]}) >>> trim_df = df.select(dfn.functions.ltrim(dfn.col("a")).alias("trimmed")) >>> trim_df.collect_column("trimmed")[0].as_py() 'a ' """ return Expr(f.ltrim(arg.expr)) def md5(arg: Expr) -> Expr: """Computes an MD5 128-bit checksum for a string expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.md5(dfn.col("a")).alias("md5")) >>> result.collect_column("md5")[0].as_py() '5d41402abc4b2a76b9719d911017c592' """ return Expr(f.md5(arg.expr)) def nanvl(x: Expr, y: Expr) -> Expr: """Returns ``x`` if ``x`` is not ``NaN``. Otherwise returns ``y``. Args: x: Expression to return when it is not NaN. y: Fallback expression to return when ``x`` is NaN. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [np.nan, 1.0], "b": [0.0, 0.0]}) >>> nanvl_df = df.select( ... dfn.functions.nanvl(dfn.col("a"), dfn.col("b")).alias("nanvl")) >>> nanvl_df.collect_column("nanvl")[0].as_py() 0.0 >>> nanvl_df.collect_column("nanvl")[1].as_py() 1.0 """ return Expr(f.nanvl(x.expr, y.expr)) def nvl(x: Expr, y: Expr) -> Expr: """Returns ``x`` if ``x`` is not ``NULL``. Otherwise returns ``y``. Args: x: Expression to return when it is not NULL. y: Fallback expression to return when ``x`` is NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [None, 1], "b": [0, 0]}) >>> nvl_df = df.select( ... dfn.functions.nvl(dfn.col("a"), dfn.col("b")).alias("nvl") ... ) >>> nvl_df.collect_column("nvl")[0].as_py() 0 >>> nvl_df.collect_column("nvl")[1].as_py() 1 """ return Expr(f.nvl(x.expr, y.expr)) def nvl2(x: Expr, y: Expr, z: Expr) -> Expr: """Returns ``y`` if ``x`` is not NULL. Otherwise returns ``z``. Args: x: Expression to check for NULL. y: Expression to return when ``x`` is not NULL. z: Expression to return when ``x`` is NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [None, 1], "b": [10, 20], "c": [30, 40]}) >>> result = df.select( ... dfn.functions.nvl2( ... dfn.col("a"), dfn.col("b"), dfn.col("c")).alias("nvl2") ... ) >>> result.collect_column("nvl2")[0].as_py() 30 >>> result.collect_column("nvl2")[1].as_py() 20 """ return Expr(f.nvl2(x.expr, y.expr, z.expr)) def octet_length(arg: Expr) -> Expr: """Returns the number of bytes of a string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.octet_length(dfn.col("a")).alias("len")) >>> result.collect_column("len")[0].as_py() 5 """ return Expr(f.octet_length(arg.expr)) def overlay( string: Expr, substring: Expr | str, start: Expr | int, length: Expr | int | None = None, ) -> Expr: """Replace a substring with a new substring. Replace the substring of string that starts at the ``start``'th character and extends for ``length`` characters with new substring. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["abcdef"]}) >>> result = df.select( ... dfn.functions.overlay(dfn.col("a"), "XY", 3, 2).alias("o")) >>> result.collect_column("o")[0].as_py() 'abXYef' """ substring = coerce_to_expr(substring) start = coerce_to_expr(start) if length is None: return Expr(f.overlay(string.expr, substring.expr, start.expr)) length = coerce_to_expr(length) return Expr(f.overlay(string.expr, substring.expr, start.expr, length.expr)) def pi() -> Expr: """Returns an approximate value of π. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> from math import pi >>> result = df.select( ... dfn.functions.pi().alias("pi") ... ) >>> result.collect_column("pi")[0].as_py() == pi True """ return Expr(f.pi()) def position(string: Expr, substring: Expr | str) -> Expr: """Finds the position from where the ``substring`` matches the ``string``. See Also: This is an alias for :py:func:`strpos`. """ return strpos(string, substring) def power(base: Expr, exponent: Expr | int | float) -> Expr: # noqa: PYI041 """Returns ``base`` raised to the power of ``exponent``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [2.0]}) >>> result = df.select( ... dfn.functions.power(dfn.col("a"), 3.0).alias("pow") ... ) >>> result.collect_column("pow")[0].as_py() 8.0 """ exponent = coerce_to_expr(exponent) return Expr(f.power(base.expr, exponent.expr)) def pow(base: Expr, exponent: Expr | int | float) -> Expr: # noqa: PYI041 """Returns ``base`` raised to the power of ``exponent``. See Also: This is an alias of :py:func:`power`. """ return power(base, exponent) def radians(arg: Expr) -> Expr: """Converts the argument from degrees to radians. Examples: >>> from math import pi >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [180.0]}) >>> result = df.select( ... dfn.functions.radians(dfn.col("a")).alias("rad") ... ) >>> result.collect_column("rad")[0].as_py() == pi True """ return Expr(f.radians(arg.expr)) def regexp_like( string: Expr, regex: Expr | str, flags: Expr | str | None = None ) -> Expr: r"""Find if any regular expression (regex) matches exist. Tests a string using a regular expression returning true if at least one match, false otherwise. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello123"]}) >>> result = df.select( ... dfn.functions.regexp_like(dfn.col("a"), "\\d+").alias("m") ... ) >>> result.collect_column("m")[0].as_py() True Use ``flags`` for case-insensitive matching: >>> result = df.select( ... dfn.functions.regexp_like( ... dfn.col("a"), "HELLO", flags="i", ... ).alias("m") ... ) >>> result.collect_column("m")[0].as_py() True """ regex = coerce_to_expr(regex) flags = coerce_to_expr_or_none(flags) return Expr( f.regexp_like( string.expr, regex.expr, flags.expr if flags is not None else None ) ) def regexp_match( string: Expr, regex: Expr | str, flags: Expr | str | None = None ) -> Expr: r"""Perform regular expression (regex) matching. Returns an array with each element containing the leftmost-first match of the corresponding index in ``regex`` to string in ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello 42 world"]}) >>> result = df.select( ... dfn.functions.regexp_match(dfn.col("a"), "(\\d+)").alias("m") ... ) >>> result.collect_column("m")[0].as_py() ['42'] Use ``flags`` for case-insensitive matching: >>> result = df.select( ... dfn.functions.regexp_match( ... dfn.col("a"), "(HELLO)", flags="i", ... ).alias("m") ... ) >>> result.collect_column("m")[0].as_py() ['hello'] """ regex = coerce_to_expr(regex) flags = coerce_to_expr_or_none(flags) return Expr( f.regexp_match( string.expr, regex.expr, flags.expr if flags is not None else None ) ) def regexp_replace( string: Expr, pattern: Expr | str, replacement: Expr | str, flags: Expr | str | None = None, ) -> Expr: r"""Replaces substring(s) matching a PCRE-like regular expression. The full list of supported features and syntax can be found at Supported flags with the addition of 'g' can be found at Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello 42"]}) >>> result = df.select( ... dfn.functions.regexp_replace( ... dfn.col("a"), "\\d+", "XX" ... ).alias("r") ... ) >>> result.collect_column("r")[0].as_py() 'hello XX' Use the ``g`` flag to replace all occurrences: >>> df = ctx.from_pydict({"a": ["a1 b2 c3"]}) >>> result = df.select( ... dfn.functions.regexp_replace( ... dfn.col("a"), "\\d+", "X", flags="g", ... ).alias("r") ... ) >>> result.collect_column("r")[0].as_py() 'aX bX cX' """ pattern = coerce_to_expr(pattern) replacement = coerce_to_expr(replacement) flags = coerce_to_expr_or_none(flags) return Expr( f.regexp_replace( string.expr, pattern.expr, replacement.expr, flags.expr if flags is not None else None, ) ) def regexp_count( string: Expr, pattern: Expr | str, start: Expr | int | None = None, flags: Expr | str | None = None, ) -> Expr: """Returns the number of matches in a string. Optional start position (the first position is 1) to search for the regular expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["abcabc"]}) >>> result = df.select( ... dfn.functions.regexp_count(dfn.col("a"), "abc").alias("c")) >>> result.collect_column("c")[0].as_py() 2 Use ``start`` to begin searching from a position, and ``flags`` for case-insensitive matching: >>> result = df.select( ... dfn.functions.regexp_count( ... dfn.col("a"), "ABC", start=4, flags="i", ... ).alias("c")) >>> result.collect_column("c")[0].as_py() 1 """ pattern = coerce_to_expr(pattern) start = coerce_to_expr_or_none(start) flags = coerce_to_expr_or_none(flags) return Expr( f.regexp_count( string.expr, pattern.expr, start.expr if start is not None else None, flags.expr if flags is not None else None, ) ) def regexp_instr( values: Expr, regex: Expr | str, start: Expr | int | None = None, n: Expr | int | None = None, flags: Expr | str | None = None, sub_expr: Expr | int | None = None, ) -> Expr: r"""Returns the position of a regular expression match in a string. Args: values: Data to search for the regular expression match. regex: Regular expression to search for. start: Optional position to start the search (the first position is 1). n: Optional occurrence of the match to find (the first occurrence is 1). flags: Optional regular expression flags to control regex behavior. sub_expr: Optionally capture group position instead of the entire match. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello 42 world"]}) >>> result = df.select( ... dfn.functions.regexp_instr(dfn.col("a"), "\\d+").alias("pos") ... ) >>> result.collect_column("pos")[0].as_py() 7 Use ``start`` to search from a position, ``n`` for the nth occurrence, and ``flags`` for case-insensitive mode: >>> df = ctx.from_pydict({"a": ["abc ABC abc"]}) >>> result = df.select( ... dfn.functions.regexp_instr( ... dfn.col("a"), "abc", ... start=2, n=1, flags="i", ... ).alias("pos") ... ) >>> result.collect_column("pos")[0].as_py() 5 Use ``sub_expr`` to get the position of a capture group: >>> result = df.select( ... dfn.functions.regexp_instr( ... dfn.col("a"), "(abc)", sub_expr=1, ... ).alias("pos") ... ) >>> result.collect_column("pos")[0].as_py() 1 """ regex = coerce_to_expr(regex) start = coerce_to_expr_or_none(start) n = coerce_to_expr_or_none(n) flags = coerce_to_expr_or_none(flags) sub_expr = coerce_to_expr_or_none(sub_expr) return Expr( f.regexp_instr( values.expr, regex.expr, start.expr if start is not None else None, n.expr if n is not None else None, flags.expr if flags is not None else None, sub_expr.expr if sub_expr is not None else None, ) ) def repeat(string: Expr, n: Expr | int) -> Expr: """Repeats the ``string`` to ``n`` times. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["ha"]}) >>> result = df.select( ... dfn.functions.repeat(dfn.col("a"), 3).alias("r")) >>> result.collect_column("r")[0].as_py() 'hahaha' """ n = coerce_to_expr(n) return Expr(f.repeat(string.expr, n.expr)) def replace(string: Expr, from_val: Expr | str, to_val: Expr | str) -> Expr: """Replaces all occurrences of ``from_val`` with ``to_val`` in the ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello world"]}) >>> result = df.select( ... dfn.functions.replace(dfn.col("a"), "world", "there").alias("r")) >>> result.collect_column("r")[0].as_py() 'hello there' """ from_val = coerce_to_expr(from_val) to_val = coerce_to_expr(to_val) return Expr(f.replace(string.expr, from_val.expr, to_val.expr)) def reverse(arg: Expr) -> Expr: """Reverse the string argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.reverse(dfn.col("a")).alias("r")) >>> result.collect_column("r")[0].as_py() 'olleh' """ return Expr(f.reverse(arg.expr)) def right(string: Expr, n: Expr | int) -> Expr: """Returns the last ``n`` characters in the ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.right(dfn.col("a"), 3).alias("r")) >>> result.collect_column("r")[0].as_py() 'llo' """ n = coerce_to_expr(n) return Expr(f.right(string.expr, n.expr)) def round(value: Expr, decimal_places: Expr | int | None = None) -> Expr: """Round the argument to the nearest integer. If the optional ``decimal_places`` is specified, round to the nearest number of decimal places. You can specify a negative number of decimal places. For example ``round(lit(125.2345), -2)`` would yield a value of ``100.0``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.567]}) >>> result = df.select(dfn.functions.round(dfn.col("a"), 2).alias("r")) >>> result.collect_column("r")[0].as_py() 1.57 """ decimal_places = coerce_to_expr(decimal_places if decimal_places is not None else 0) return Expr(f.round(value.expr, decimal_places.expr)) def rpad(string: Expr, count: Expr | int, characters: Expr | str | None = None) -> Expr: """Add right padding to a string. Extends the string to length length by appending the characters fill (a space by default). If the string is already longer than length then it is truncated. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hi"]}) >>> result = df.select( ... dfn.functions.rpad(dfn.col("a"), 5, "!").alias("r")) >>> result.collect_column("r")[0].as_py() 'hi!!!' """ count = coerce_to_expr(count) characters = coerce_to_expr(characters if characters is not None else " ") return Expr(f.rpad(string.expr, count.expr, characters.expr)) def rtrim(arg: Expr) -> Expr: """Removes all characters, spaces by default, from the end of a string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [" a "]}) >>> trim_df = df.select(dfn.functions.rtrim(dfn.col("a")).alias("trimmed")) >>> trim_df.collect_column("trimmed")[0].as_py() ' a' """ return Expr(f.rtrim(arg.expr)) def sha224(arg: Expr) -> Expr: """Computes the SHA-224 hash of a binary string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.sha224(dfn.col("a")).alias("h") ... ) >>> result.collect_column("h")[0].as_py().hex() 'ea09ae9cc6768c50fcee903ed054556e5bfc8347907f12598aa24193' """ return Expr(f.sha224(arg.expr)) def sha256(arg: Expr) -> Expr: """Computes the SHA-256 hash of a binary string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.sha256(dfn.col("a")).alias("h") ... ) >>> result.collect_column("h")[0].as_py().hex() '2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824' """ return Expr(f.sha256(arg.expr)) def sha384(arg: Expr) -> Expr: """Computes the SHA-384 hash of a binary string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.sha384(dfn.col("a")).alias("h") ... ) >>> result.collect_column("h")[0].as_py().hex() '59e1748777448c69de6b800d7a33bbfb9ff1b... """ return Expr(f.sha384(arg.expr)) def sha512(arg: Expr) -> Expr: """Computes the SHA-512 hash of a binary string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.sha512(dfn.col("a")).alias("h") ... ) >>> result.collect_column("h")[0].as_py().hex() '9b71d224bd62f3785d96d46ad3ea3d73319bfb... """ return Expr(f.sha512(arg.expr)) def signum(arg: Expr) -> Expr: """Returns the sign of the argument (-1, 0, +1). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [-5.0, 0.0, 5.0]}) >>> result = df.select(dfn.functions.signum(dfn.col("a")).alias("s")) >>> result.collect_column("s").to_pylist() [-1.0, 0.0, 1.0] """ return Expr(f.signum(arg.expr)) def sin(arg: Expr) -> Expr: """Returns the sine of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.sin(dfn.col("a")).alias("sin")) >>> result.collect_column("sin")[0].as_py() 0.0 """ return Expr(f.sin(arg.expr)) def sinh(arg: Expr) -> Expr: """Returns the hyperbolic sine of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.sinh(dfn.col("a")).alias("sinh")) >>> result.collect_column("sinh")[0].as_py() 0.0 """ return Expr(f.sinh(arg.expr)) def split_part(string: Expr, delimiter: Expr | str, index: Expr | int) -> Expr: """Split a string and return one part. Splits a string based on a delimiter and picks out the desired field based on the index. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["a,b,c"]}) >>> result = df.select( ... dfn.functions.split_part(dfn.col("a"), ",", 2).alias("s")) >>> result.collect_column("s")[0].as_py() 'b' """ delimiter = coerce_to_expr(delimiter) index = coerce_to_expr(index) return Expr(f.split_part(string.expr, delimiter.expr, index.expr)) def sqrt(arg: Expr) -> Expr: """Returns the square root of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [9.0]}) >>> result = df.select(dfn.functions.sqrt(dfn.col("a")).alias("sqrt")) >>> result.collect_column("sqrt")[0].as_py() 3.0 """ return Expr(f.sqrt(arg.expr)) def starts_with(string: Expr, prefix: Expr | str) -> Expr: """Returns true if string starts with prefix. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello_from_datafusion"]}) >>> result = df.select( ... dfn.functions.starts_with(dfn.col("a"), "hello").alias("sw")) >>> result.collect_column("sw")[0].as_py() True """ prefix = coerce_to_expr(prefix) return Expr(f.starts_with(string.expr, prefix.expr)) def strpos(string: Expr, substring: Expr | str) -> Expr: """Finds the position from where the ``substring`` matches the ``string``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.strpos(dfn.col("a"), "llo").alias("pos")) >>> result.collect_column("pos")[0].as_py() 3 """ substring = coerce_to_expr(substring) return Expr(f.strpos(string.expr, substring.expr)) def substr(string: Expr, position: Expr | int) -> Expr: """Substring from the ``position`` to the end. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.substr(dfn.col("a"), 3).alias("s")) >>> result.collect_column("s")[0].as_py() 'llo' """ position = coerce_to_expr(position) return Expr(f.substr(string.expr, position.expr)) def substr_index(string: Expr, delimiter: Expr | str, count: Expr | int) -> Expr: """Returns an indexed substring. The return will be the ``string`` from before ``count`` occurrences of ``delimiter``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["a.b.c"]}) >>> result = df.select( ... dfn.functions.substr_index(dfn.col("a"), ".", 2).alias("s")) >>> result.collect_column("s")[0].as_py() 'a.b' """ delimiter = coerce_to_expr(delimiter) count = coerce_to_expr(count) return Expr(f.substr_index(string.expr, delimiter.expr, count.expr)) def substring(string: Expr, position: Expr | int, length: Expr | int) -> Expr: """Substring from the ``position`` with ``length`` characters. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello world"]}) >>> result = df.select( ... dfn.functions.substring(dfn.col("a"), 1, 5).alias("s")) >>> result.collect_column("s")[0].as_py() 'hello' """ position = coerce_to_expr(position) length = coerce_to_expr(length) return Expr(f.substring(string.expr, position.expr, length.expr)) def tan(arg: Expr) -> Expr: """Returns the tangent of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.tan(dfn.col("a")).alias("tan")) >>> result.collect_column("tan")[0].as_py() 0.0 """ return Expr(f.tan(arg.expr)) def tanh(arg: Expr) -> Expr: """Returns the hyperbolic tangent of the argument. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0]}) >>> result = df.select(dfn.functions.tanh(dfn.col("a")).alias("tanh")) >>> result.collect_column("tanh")[0].as_py() 0.0 """ return Expr(f.tanh(arg.expr)) def to_hex(arg: Expr) -> Expr: """Converts an integer to a hexadecimal string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [255]}) >>> result = df.select(dfn.functions.to_hex(dfn.col("a")).alias("hex")) >>> result.collect_column("hex")[0].as_py() 'ff' """ return Expr(f.to_hex(arg.expr)) def now() -> Expr: """Returns the current timestamp in nanoseconds. This will use the same value for all instances of now() in same statement. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.now().alias("now") ... ) Use .value instead of .as_py() because nanosecond timestamps require pandas to convert to Python datetime objects. >>> result.collect_column("now")[0].value > 0 True """ return Expr(f.now()) def current_timestamp() -> Expr: """Returns the current timestamp in nanoseconds. See Also: This is an alias for :py:func:`now`. """ return now() def to_char(arg: Expr, formatter: Expr | str) -> Expr: """Returns a string representation of a date, time, timestamp or duration. For usage of ``formatter`` see the rust chrono package ``strftime`` package. [Documentation here.](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-01-01T00:00:00"]}) >>> result = df.select( ... dfn.functions.to_char( ... dfn.functions.to_timestamp(dfn.col("a")), ... "%Y/%m/%d", ... ).alias("formatted") ... ) >>> result.collect_column("formatted")[0].as_py() '2021/01/01' """ formatter = coerce_to_expr(formatter) return Expr(f.to_char(arg.expr, formatter.expr)) def date_format(arg: Expr, formatter: Expr | str) -> Expr: """Returns a string representation of a date, time, timestamp or duration. See Also: This is an alias for :py:func:`to_char`. """ return to_char(arg, formatter) def _unwrap_exprs(args: Iterable[Expr]) -> list: return [arg.expr for arg in args] def to_date(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a value to a date (YYYY-MM-DD). Supports strings, numeric and timestamp types as input. Integers and doubles are interpreted as days since the unix epoch. Strings are parsed as YYYY-MM-DD (e.g. '2023-07-20') if ``formatters`` are not provided. For usage of ``formatters`` see the rust chrono package ``strftime`` package. [Documentation here.](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-07-20"]}) >>> result = df.select( ... dfn.functions.to_date(dfn.col("a")).alias("dt")) >>> str(result.collect_column("dt")[0].as_py()) '2021-07-20' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["20-07-2021"]}) >>> result = df.select( ... dfn.functions.to_date(dfn.col("a"), "%d-%m-%Y").alias("dt")) >>> str(result.collect_column("dt")[0].as_py()) '2021-07-20' """ return Expr(f.to_date(arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters)))) def to_local_time(*args: Expr) -> Expr: """Converts a timestamp with a timezone to a timestamp without a timezone. This function handles daylight saving time changes. """ return Expr(f.to_local_time(*_unwrap_exprs(args))) def to_time(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a value to a time. Supports strings and timestamps as input. If ``formatters`` is not provided strings are parsed as HH:MM:SS, HH:MM or HH:MM:SS.nnnnnnnnn; For usage of ``formatters`` see the rust chrono package ``strftime`` package. [Documentation here.](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["14:30:00"]}) >>> result = df.select( ... dfn.functions.to_time(dfn.col("a")).alias("t")) >>> str(result.collect_column("t")[0].as_py()) '14:30:00' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["14h30m00s"]}) >>> result = df.select( ... dfn.functions.to_time(dfn.col("a"), "%Hh%Mm%Ss").alias("t")) >>> str(result.collect_column("t")[0].as_py()) '14:30:00' """ return Expr(f.to_time(arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters)))) def to_timestamp(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a string and optional formats to a ``Timestamp`` in nanoseconds. For usage of ``formatters`` see the rust chrono package ``strftime`` package. [Documentation here.](https://docs.rs/chrono/latest/chrono/format/strftime/index.html) Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-01-01T00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp( ... dfn.col("a") ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["01/01/2021 00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp( ... dfn.col("a"), "%d/%m/%Y %H:%M:%S" ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' """ return Expr( f.to_timestamp(arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters))) ) def to_timestamp_millis(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a string and optional formats to a ``Timestamp`` in milliseconds. See :py:func:`to_timestamp` for a description on how to use formatters. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-01-01T00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_millis( ... dfn.col("a") ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["01/01/2021 00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_millis( ... dfn.col("a"), "%d/%m/%Y %H:%M:%S" ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' """ return Expr( f.to_timestamp_millis(arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters))) ) def to_timestamp_micros(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a string and optional formats to a ``Timestamp`` in microseconds. See :py:func:`to_timestamp` for a description on how to use formatters. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-01-01T00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_micros( ... dfn.col("a") ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["01/01/2021 00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_micros( ... dfn.col("a"), "%d/%m/%Y %H:%M:%S" ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' """ return Expr( f.to_timestamp_micros(arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters))) ) def to_timestamp_nanos(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a string and optional formats to a ``Timestamp`` in nanoseconds. See :py:func:`to_timestamp` for a description on how to use formatters. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-01-01T00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_nanos( ... dfn.col("a") ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["01/01/2021 00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_nanos( ... dfn.col("a"), "%d/%m/%Y %H:%M:%S" ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' """ return Expr( f.to_timestamp_nanos(arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters))) ) def to_timestamp_seconds(arg: Expr, *formatters: Expr | str) -> Expr: """Converts a string and optional formats to a ``Timestamp`` in seconds. See :py:func:`to_timestamp` for a description on how to use formatters. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-01-01T00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_seconds( ... dfn.col("a") ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["01/01/2021 00:00:00"]}) >>> result = df.select( ... dfn.functions.to_timestamp_seconds( ... dfn.col("a"), "%d/%m/%Y %H:%M:%S" ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '2021-01-01 00:00:00' """ return Expr( f.to_timestamp_seconds( arg.expr, *_unwrap_exprs(coerce_to_expr_list(formatters)) ) ) def to_unixtime(string: Expr, *format_arguments: Expr | str) -> Expr: """Converts a string and optional formats to a Unixtime. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["1970-01-01T00:00:00"]}) >>> result = df.select(dfn.functions.to_unixtime(dfn.col("a")).alias("u")) >>> result.collect_column("u")[0].as_py() 0 Pass a format string as a bare ``str``: >>> df = ctx.from_pydict({"a": ["01/01/1970 00:00:00"]}) >>> result = df.select( ... dfn.functions.to_unixtime( ... dfn.col("a"), "%d/%m/%Y %H:%M:%S" ... ).alias("u") ... ) >>> result.collect_column("u")[0].as_py() 0 """ return Expr( f.to_unixtime( string.expr, *_unwrap_exprs(coerce_to_expr_list(format_arguments)) ) ) def current_date() -> Expr: """Returns current UTC date as a Date32 value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.current_date().alias("d") ... ) >>> result.collect_column("d")[0].as_py() is not None True """ return Expr(f.current_date()) today = current_date def current_time() -> Expr: """Returns current UTC time as a Time64 value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.current_time().alias("t") ... ) Use .value instead of .as_py() because nanosecond timestamps require pandas to convert to Python datetime objects. >>> result.collect_column("t")[0].value > 0 True """ return Expr(f.current_time()) def datepart(part: Expr | str, date: Expr) -> Expr: """Return a specified part of a date. See Also: This is an alias for :py:func:`date_part`. """ return _date_part(part, date, "datepart") def date_part(part: Expr | str, date: Expr) -> Expr: """Extracts a subfield from the date. Args: part: The part of the date to extract. Must be one of ``"year"``, ``"month"``, ``"day"``, ``"hour"``, ``"minute"``, ``"second"``, etc. date: The date expression to extract from. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-07-15T00:00:00"]}) >>> df = df.select(dfn.functions.to_timestamp(dfn.col("a")).alias("a")) >>> result = df.select( ... dfn.functions.date_part("year", dfn.col("a")).alias("y")) >>> result.collect_column("y")[0].as_py() 2021 """ return _date_part(part, date, "date_part") def _date_part(part: Expr | str, date: Expr, function_name: str) -> Expr: _warn_if_expr_for_literal_arg(part, function_name, "part") part = coerce_to_expr(part) return Expr(f.date_part(part.expr, date.expr)) def extract(part: Expr | str, date: Expr) -> Expr: """Extracts a subfield from the date. See Also: This is an alias for :py:func:`date_part`. """ return _date_part(part, date, "extract") def date_trunc(part: Expr | str, date: Expr) -> Expr: """Truncates the date to a specified level of precision. Args: part: The precision to truncate to. Must be one of ``"year"``, ``"month"``, ``"day"``, ``"hour"``, ``"minute"``, ``"second"``, etc. date: The date expression to truncate. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["2021-07-15T12:34:56"]}) >>> df = df.select(dfn.functions.to_timestamp(dfn.col("a")).alias("a")) >>> result = df.select( ... dfn.functions.date_trunc("month", dfn.col("a")).alias("t") ... ) >>> str(result.collect_column("t")[0].as_py()) '2021-07-01 00:00:00' """ return _date_trunc(part, date, "date_trunc") def _date_trunc(part: Expr | str, date: Expr, function_name: str) -> Expr: _warn_if_expr_for_literal_arg(part, function_name, "part") part = coerce_to_expr(part) return Expr(f.date_trunc(part.expr, date.expr)) def datetrunc(part: Expr | str, date: Expr) -> Expr: """Truncates the date to a specified level of precision. See Also: This is an alias for :py:func:`date_trunc`. """ return _date_trunc(part, date, "datetrunc") def date_bin(stride: Expr | str, source: Expr | str, origin: Expr | str) -> Expr: """Coerces an arbitrary timestamp to the start of the nearest specified interval. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"timestamp": ['2021-07-15 12:34:56', '2021-01-01']}) >>> result = df.select( ... dfn.functions.date_bin( ... "15 minutes", ... dfn.col("timestamp"), ... "2001-01-01 00:00:00", ... ).alias("b") ... ) >>> str(result.collect_column("b")[0].as_py()) '2021-07-15 12:30:00' >>> str(result.collect_column("b")[1].as_py()) '2021-01-01 00:00:00' ``source`` may also be a bare literal: >>> result = df.select( ... dfn.functions.date_bin( ... "15 minutes", "2021-07-15 12:34:56", "2001-01-01 00:00:00" ... ).alias("b") ... ) >>> str(result.collect_column("b")[0].as_py()) '2021-07-15 12:30:00' """ # date_bin's planner coerces Utf8 (not Utf8View) literals to Interval/Timestamp, # so wrap bare strs via string_literal to force Utf8. stride = Expr.string_literal(stride) if isinstance(stride, str) else stride source = Expr.string_literal(source) if isinstance(source, str) else source origin = Expr.string_literal(origin) if isinstance(origin, str) else origin return Expr(f.date_bin(stride.expr, source.expr, origin.expr)) def make_date(year: Expr | int, month: Expr | int, day: Expr | int) -> Expr: """Make a date from year, month and day component parts. Examples: >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [2024], "m": [1], "d": [15]}) >>> result = df.select( ... dfn.functions.make_date(dfn.col("y"), dfn.col("m"), ... dfn.col("d")).alias("dt")) >>> result.collect_column("dt")[0].as_py() datetime.date(2024, 1, 15) Pass bare ints for any component: >>> df = ctx.from_pydict({"y": [2024]}) >>> result = df.select( ... dfn.functions.make_date(dfn.col("y"), 1, 15).alias("dt")) >>> result.collect_column("dt")[0].as_py() datetime.date(2024, 1, 15) """ year = coerce_to_expr(year) month = coerce_to_expr(month) day = coerce_to_expr(day) return Expr(f.make_date(year.expr, month.expr, day.expr)) def make_time(hour: Expr | int, minute: Expr | int, second: Expr | int) -> Expr: """Make a time from hour, minute and second component parts. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"h": [12], "m": [30], "s": [0]}) >>> result = df.select( ... dfn.functions.make_time(dfn.col("h"), dfn.col("m"), ... dfn.col("s")).alias("t")) >>> result.collect_column("t")[0].as_py() datetime.time(12, 30) Pass bare ints for any component: >>> df = ctx.from_pydict({"h": [12]}) >>> result = df.select( ... dfn.functions.make_time(dfn.col("h"), 30, 0).alias("t")) >>> result.collect_column("t")[0].as_py() datetime.time(12, 30) """ hour = coerce_to_expr(hour) minute = coerce_to_expr(minute) second = coerce_to_expr(second) return Expr(f.make_time(hour.expr, minute.expr, second.expr)) def translate(string: Expr, from_val: Expr | str, to_val: Expr | str) -> Expr: """Replaces the characters in ``from_val`` with the counterpart in ``to_val``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select( ... dfn.functions.translate(dfn.col("a"), "helo", "HELO").alias("t")) >>> result.collect_column("t")[0].as_py() 'HELLO' """ from_val = coerce_to_expr(from_val) to_val = coerce_to_expr(to_val) return Expr(f.translate(string.expr, from_val.expr, to_val.expr)) def trim(arg: Expr) -> Expr: """Removes all characters, spaces by default, from both sides of a string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [" hello "]}) >>> result = df.select(dfn.functions.trim(dfn.col("a")).alias("t")) >>> result.collect_column("t")[0].as_py() 'hello' """ return Expr(f.trim(arg.expr)) def trunc(num: Expr, precision: Expr | int | None = None) -> Expr: """Truncate the number toward zero with optional precision. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.567]}) >>> result = df.select( ... dfn.functions.trunc(dfn.col("a")).alias("t")) >>> result.collect_column("t")[0].as_py() 1.0 >>> result = df.select( ... dfn.functions.trunc(dfn.col("a"), precision=2).alias("t")) >>> result.collect_column("t")[0].as_py() 1.56 """ if precision is not None: precision = coerce_to_expr(precision) return Expr(f.trunc(num.expr, precision.expr)) return Expr(f.trunc(num.expr)) def upper(arg: Expr) -> Expr: """Converts a string to uppercase. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello"]}) >>> result = df.select(dfn.functions.upper(dfn.col("a")).alias("u")) >>> result.collect_column("u")[0].as_py() 'HELLO' """ return Expr(f.upper(arg.expr)) def make_array(*args: Expr) -> Expr: """Returns an array using the specified input expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.make_array( ... dfn.lit(1), dfn.lit(2), dfn.lit(3) ... ).alias("arr")) >>> result.collect_column("arr")[0].as_py() [1, 2, 3] """ args = [arg.expr for arg in args] return Expr(f.make_array(args)) def make_list(*args: Expr) -> Expr: """Returns an array using the specified input expressions. See Also: This is an alias for :py:func:`make_array`. """ return make_array(*args) def array(*args: Expr) -> Expr: """Returns an array using the specified input expressions. See Also: This is an alias for :py:func:`make_array`. """ return make_array(*args) def range(start: Expr, stop: Expr, step: Expr) -> Expr: """Create a list of values in the range between start and stop. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.range(dfn.lit(0), dfn.lit(5), dfn.lit(2)).alias("r")) >>> result.collect_column("r")[0].as_py() [0, 2, 4] """ return Expr(f.range(start.expr, stop.expr, step.expr)) def uuid() -> Expr: """Returns uuid v4 as a string value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.uuid().alias("u") ... ) >>> len(result.collect_column("u")[0].as_py()) == 36 True """ return Expr(f.uuid()) def struct(*args: Expr) -> Expr: """Returns a struct with the given arguments. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1], "b": [2]}) >>> result = df.select( ... dfn.functions.struct( ... dfn.col("a"), dfn.col("b") ... ).alias("s") ... ) Children in the new struct will always be `c0`, ..., `cN-1` for `N` children. >>> result.collect_column("s")[0].as_py() == {"c0": 1, "c1": 2} True """ args = [arg.expr for arg in args] return Expr(f.struct(*args)) def named_struct(name_pairs: list[tuple[str, Expr]]) -> Expr: """Returns a struct with the given names and arguments pairs. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.named_struct( ... [("x", dfn.lit(10)), ("y", dfn.lit(20))] ... ).alias("s") ... ) >>> result.collect_column("s")[0].as_py() == {"x": 10, "y": 20} True """ name_pair_exprs = [ [Expr.literal(pa.scalar(pair[0], type=pa.string())), pair[1]] for pair in name_pairs ] # flatten name_pairs = [x.expr for xs in name_pair_exprs for x in xs] return Expr(f.named_struct(*name_pairs)) def from_unixtime(arg: Expr) -> Expr: """Converts an integer to RFC3339 timestamp format string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0]}) >>> result = df.select( ... dfn.functions.from_unixtime( ... dfn.col("a") ... ).alias("ts") ... ) >>> str(result.collect_column("ts")[0].as_py()) '1970-01-01 00:00:00' """ return Expr(f.from_unixtime(arg.expr)) def arrow_typeof(arg: Expr) -> Expr: """Returns the Arrow type of the expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select(dfn.functions.arrow_typeof(dfn.col("a")).alias("t")) >>> result.collect_column("t")[0].as_py() 'Int64' """ return Expr(f.arrow_typeof(arg.expr)) def arrow_cast(expr: Expr, data_type: Expr | str | pa.DataType) -> Expr: """Casts an expression to a specified data type. The ``data_type`` can be a string, a ``pyarrow.DataType``, or an ``Expr``. For simple types, :py:meth:`Expr.cast() ` is more concise (e.g., ``col("a").cast(pa.float64())``). Use ``arrow_cast`` when you want to specify the target type as a string using DataFusion's type syntax, which can be more readable for complex types like ``"Timestamp(Nanosecond, None)"``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.arrow_cast(dfn.col("a"), "Float64").alias("c") ... ) >>> result.collect_column("c")[0].as_py() 1.0 >>> result = df.select( ... dfn.functions.arrow_cast( ... dfn.col("a"), data_type=pa.float64() ... ).alias("c") ... ) >>> result.collect_column("c")[0].as_py() 1.0 """ _warn_if_expr_for_literal_arg(data_type, "arrow_cast", "data_type") if isinstance(data_type, pa.DataType): return expr.cast(data_type) if isinstance(data_type, str): data_type = Expr.string_literal(data_type) return Expr(f.arrow_cast(expr.expr, data_type.expr)) def arrow_try_cast(expr: Expr, data_type: Expr | str | pa.DataType) -> Expr: """Casts an expression to a specified data type, returning NULL on failure. Like :py:func:`arrow_cast` but produces NULL instead of erroring when the cast cannot be performed. The ``data_type`` may be a string in DataFusion type syntax (for example ``"Float64"``), a ``pyarrow.DataType``, or an ``Expr`` of string type. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["oops"]}) >>> result = df.select( ... dfn.functions.arrow_try_cast(dfn.col("a"), "Float64").alias("c") ... ) >>> result.collect_column("c")[0].as_py() is None True >>> result = df.select( ... dfn.functions.arrow_try_cast( ... dfn.col("a"), data_type=pa.float64() ... ).alias("c") ... ) >>> result.collect_column("c")[0].as_py() is None True """ _warn_if_expr_for_literal_arg(data_type, "arrow_try_cast", "data_type") if isinstance(data_type, pa.DataType): return expr.try_cast(data_type) if isinstance(data_type, str): data_type = Expr.string_literal(data_type) return Expr(f.arrow_try_cast(expr.expr, data_type.expr)) def arrow_field(expr: Expr) -> Expr: """Returns the Arrow field information of an expression as a struct. The returned struct contains the field's name, data type, nullability, and metadata. Examples: >>> field = pa.field("val", pa.int64(), metadata={"k": "v"}) >>> schema = pa.schema([field]) >>> batch = pa.RecordBatch.from_arrays([pa.array([1])], schema=schema) >>> ctx = dfn.SessionContext() >>> df = ctx.create_dataframe([[batch]]) >>> result = df.select( ... dfn.functions.arrow_field(dfn.col("val")).alias("f") ... ) >>> out = result.collect_column("f")[0].as_py() >>> out["name"], out["data_type"], out["nullable"], out["metadata"] ('val', 'Int64', True, [('k', 'v')]) """ return Expr(f.arrow_field(expr.expr)) def cast_to_type(value: Expr, type_ref: Expr) -> Expr: """Casts ``value`` to the data type of ``type_ref``. Only the *type* of ``type_ref`` is used; its value is ignored. This is useful when the target type comes from another column or expression rather than being known up-front. Casts that fail produce an error; use :py:func:`try_cast_to_type` for the NULL-on-failure variant. If the target type is known statically, prefer :py:func:`arrow_cast` (or :py:func:`arrow_try_cast` for the NULL-on-failure variant) and pass a type string or ``pyarrow.DataType`` directly. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1], "b": [1.0]}) >>> result = df.select( ... dfn.functions.cast_to_type( ... dfn.col("a"), dfn.col("b") ... ).alias("c") ... ) >>> result.collect_column("c")[0].as_py() 1.0 """ return Expr(f.cast_to_type(value.expr, type_ref.expr)) def try_cast_to_type(value: Expr, type_ref: Expr) -> Expr: """Casts ``value`` to the data type of ``type_ref``, NULL on failure. Like :py:func:`cast_to_type`, but casts that fail produce NULL instead of erroring. Only the *type* of ``type_ref`` is used; its value is ignored. If the target type is known statically, prefer :py:func:`arrow_try_cast` and pass a type string or ``pyarrow.DataType`` directly. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["oops"], "b": [1.0]}) >>> result = df.select( ... dfn.functions.try_cast_to_type( ... dfn.col("a"), dfn.col("b") ... ).alias("c") ... ) >>> result.collect_column("c")[0].as_py() is None True """ return Expr(f.try_cast_to_type(value.expr, type_ref.expr)) def arrow_metadata(expr: Expr, key: Expr | str | None = None) -> Expr: """Returns the metadata of the input expression. If called with one argument, returns a Map of all metadata key-value pairs. If called with two arguments, returns the value for the specified metadata key. Examples: >>> field = pa.field("val", pa.int64(), metadata={"k": "v"}) >>> schema = pa.schema([field]) >>> batch = pa.RecordBatch.from_arrays([pa.array([1])], schema=schema) >>> ctx = dfn.SessionContext() >>> df = ctx.create_dataframe([[batch]]) >>> result = df.select( ... dfn.functions.arrow_metadata(dfn.col("val")).alias("meta") ... ) >>> ("k", "v") in result.collect_column("meta")[0].as_py() True >>> result = df.select( ... dfn.functions.arrow_metadata( ... dfn.col("val"), key="k" ... ).alias("meta_val") ... ) >>> result.collect_column("meta_val")[0].as_py() 'v' """ if key is None: return Expr(f.arrow_metadata(expr.expr)) _warn_if_expr_for_literal_arg(key, "arrow_metadata", "key") if isinstance(key, str): key = Expr.string_literal(key) return Expr(f.arrow_metadata(expr.expr, key.expr)) def with_metadata(expr: Expr, metadata: dict[str, str]) -> Expr: """Attaches Arrow field metadata (key/value pairs) to the input expression. This is the inverse of :py:func:`arrow_metadata`. Existing metadata on the input field is preserved; new keys overwrite on collision. Keys must be non-empty strings; empty values are allowed. An empty ``metadata`` dict is a no-op and returns the input expression unchanged. Empty keys raise :py:class:`ValueError`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.with_metadata( ... dfn.col("a"), {"unit": "ms"} ... ).alias("a") ... ) >>> result.select( ... dfn.functions.arrow_metadata(dfn.col("a"), "unit").alias("u") ... ).collect_column("u")[0].as_py() 'ms' """ if not metadata: return expr args = [expr.expr] for k, v in metadata.items(): if not k: msg = "with_metadata keys must be non-empty strings" raise ValueError(msg) args.append(Expr.string_literal(k).expr) args.append(Expr.string_literal(v).expr) return Expr(f.with_metadata(*args)) def get_field(expr: Expr, *names: Expr | str) -> Expr: """Extracts a (possibly nested) field from a struct or map by name. Pass one name for a single-level lookup, or several names to walk a path of nested struct/map fields in a single ``get_field`` call. For a single static-string name, ``expr["field"]`` is a convenient shorthand; use ``get_field`` when the field name is a dynamic :py:class:`~datafusion.expr.Expr` or when traversing multiple levels at once. Args: expr: The struct or map expression to read from. *names: One or more field names (``str``) or expressions (:py:class:`~datafusion.expr.Expr`). Examples: Single-level lookup: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1], "b": [2]}) >>> df = df.with_column( ... "s", ... F.named_struct([("x", dfn.col("a")), ("y", dfn.col("b"))]), ... ) >>> result = df.select( ... F.get_field(dfn.col("s"), "x").alias("x_val") ... ) >>> result.collect_column("x_val")[0].as_py() 1 Equivalent using bracket syntax: >>> result = df.select( ... dfn.col("s")["x"].alias("x_val") ... ) >>> result.collect_column("x_val")[0].as_py() 1 Multi-level lookup: >>> df = df.with_column( ... "outer", ... F.named_struct([("inner", dfn.col("s"))]), ... ) >>> result = df.select( ... F.get_field(dfn.col("outer"), "inner", "x").alias("x_val") ... ) >>> result.collect_column("x_val")[0].as_py() 1 """ if not names: msg = "get_field requires at least one field name" raise ValueError(msg) resolved = [Expr.string_literal(n) if isinstance(n, str) else n for n in names] return Expr(f.get_field(expr.expr, [n.expr for n in resolved])) def union_extract(union_expr: Expr, field_name: Expr | str) -> Expr: """Extracts a value from a union type by field name. Returns the value of the named field if it is the currently selected variant, otherwise returns NULL. Examples: >>> ctx = dfn.SessionContext() >>> types = pa.array([0, 1, 0], type=pa.int8()) >>> offsets = pa.array([0, 0, 1], type=pa.int32()) >>> arr = pa.UnionArray.from_dense( ... types, offsets, [pa.array([1, 2]), pa.array(["hi"])], ... ["int", "str"], [0, 1], ... ) >>> batch = pa.RecordBatch.from_arrays([arr], names=["u"]) >>> df = ctx.create_dataframe([[batch]]) >>> result = df.select( ... dfn.functions.union_extract(dfn.col("u"), "int").alias("val") ... ) >>> result.collect_column("val").to_pylist() [1, None, 2] """ if isinstance(field_name, str): field_name = Expr.string_literal(field_name) return Expr(f.union_extract(union_expr.expr, field_name.expr)) def union_tag(union_expr: Expr) -> Expr: """Returns the tag (active field name) of a union type. Examples: >>> ctx = dfn.SessionContext() >>> types = pa.array([0, 1, 0], type=pa.int8()) >>> offsets = pa.array([0, 0, 1], type=pa.int32()) >>> arr = pa.UnionArray.from_dense( ... types, offsets, [pa.array([1, 2]), pa.array(["hi"])], ... ["int", "str"], [0, 1], ... ) >>> batch = pa.RecordBatch.from_arrays([arr], names=["u"]) >>> df = ctx.create_dataframe([[batch]]) >>> result = df.select( ... dfn.functions.union_tag(dfn.col("u")).alias("tag") ... ) >>> result.collect_column("tag").to_pylist() ['int', 'str', 'int'] """ return Expr(f.union_tag(union_expr.expr)) def version() -> Expr: """Returns the DataFusion version string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.empty_table() >>> result = df.select(dfn.functions.version().alias("v")) >>> "Apache DataFusion" in result.collect_column("v")[0].as_py() True """ return Expr(f.version()) def row(*args: Expr) -> Expr: """Returns a struct with the given arguments. See Also: This is an alias for :py:func:`struct`. """ return struct(*args) def random() -> Expr: """Returns a random value in the range ``0.0 <= x < 1.0``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.random().alias("r") ... ) >>> val = result.collect_column("r")[0].as_py() >>> 0.0 <= val < 1.0 True """ return Expr(f.random()) def array_append(array: Expr, element: Expr) -> Expr: """Appends an element to the end of an array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_append(dfn.col("a"), dfn.lit(4)).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2, 3, 4] """ return Expr(f.array_append(array.expr, element.expr)) def array_push_back(array: Expr, element: Expr) -> Expr: """Appends an element to the end of an array. See Also: This is an alias for :py:func:`array_append`. """ return array_append(array, element) def list_append(array: Expr, element: Expr) -> Expr: """Appends an element to the end of an array. See Also: This is an alias for :py:func:`array_append`. """ return array_append(array, element) def list_push_back(array: Expr, element: Expr) -> Expr: """Appends an element to the end of an array. See Also: This is an alias for :py:func:`array_append`. """ return array_append(array, element) def array_concat(*args: Expr) -> Expr: """Concatenates the input arrays. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2]], "b": [[3, 4]]}) >>> result = df.select( ... dfn.functions.array_concat(dfn.col("a"), dfn.col("b")).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2, 3, 4] """ args = [arg.expr for arg in args] return Expr(f.array_concat(args)) def array_cat(*args: Expr) -> Expr: """Concatenates the input arrays. See Also: This is an alias for :py:func:`array_concat`. """ return array_concat(*args) def array_dims(array: Expr) -> Expr: """Returns an array of the array's dimensions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select(dfn.functions.array_dims(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() [3] """ return Expr(f.array_dims(array.expr)) def array_distinct(array: Expr) -> Expr: """Returns distinct values from the array after removing duplicates. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_distinct( ... dfn.col("a") ... ).alias("result") ... ) >>> sorted( ... result.collect_column("result")[0].as_py() ... ) [1, 2, 3] """ return Expr(f.array_distinct(array.expr)) def array_compact(array: Expr) -> Expr: """Removes NULL values from the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, None, 2, None, 3]]}) >>> result = df.select( ... dfn.functions.array_compact(dfn.col("a")).alias("result") ... ) >>> result.collect_column("result")[0].as_py() [1, 2, 3] """ return Expr(f.array_compact(array.expr)) def array_normalize(array: Expr) -> Expr: """Scales a numeric array so it has Euclidean length 1. Treats the array as a vector and divides every element by the vector's Euclidean (L2) norm — the square root of the sum of the squared elements. The returned array points in the same direction as the input but has a magnitude of 1, which makes it suitable for cosine-similarity comparisons and other operations that expect unit vectors. For the input ``[3.0, 4.0]`` the L2 norm is ``sqrt(3**2 + 4**2) = 5``, so each element is divided by 5 to produce ``[0.6, 0.8]``. Normalizing the zero vector is undefined (it would divide by zero), so the function returns NULL for an all-zero input. NULL is also returned if any element of the input array is NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[3.0, 4.0]]}) >>> result = df.select( ... dfn.functions.array_normalize(dfn.col("a")).alias("result") ... ) >>> result.collect_column("result")[0].as_py() [0.6, 0.8] The zero vector has no direction to preserve, so the result is NULL: >>> df_zero = ctx.from_pydict({"a": [[0.0, 0.0]]}) >>> result = df_zero.select( ... dfn.functions.array_normalize(dfn.col("a")).alias("result") ... ) >>> result.collect_column("result")[0].as_py() is None True """ return Expr(f.array_normalize(array.expr)) def cosine_distance(array1: Expr, array2: Expr) -> Expr: """Measures how much two numeric arrays differ in direction. Treats each input as a vector and compares the angle between them, ignoring their magnitudes. The result is ``1 - cosine_similarity``, where cosine similarity is the dot product of the two vectors divided by the product of their Euclidean (L2) norms. The returned value ranges from 0 to 2: * ``0`` — vectors point in the same direction (any positive scaling of one yields the other). * ``1`` — vectors are orthogonal (no shared direction). * ``2`` — vectors point in exactly opposite directions. This is the standard distance metric for comparing embedding vectors (text, image, audio) where direction carries the meaning and overall magnitude does not. Both arrays must have the same length; otherwise execution fails. If either input is the zero vector the cosine is undefined and the function returns NULL. Examples: Identical vectors have distance ``0``: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict( ... {"a": [[1.0, 2.0, 3.0]], "b": [[1.0, 2.0, 3.0]]} ... ) >>> result = df.select( ... dfn.functions.cosine_distance( ... dfn.col("a"), dfn.col("b") ... ).alias("result") ... ) >>> result.collect_column("result")[0].as_py() 0.0 Orthogonal vectors have distance ``1``: >>> df_orth = ctx.from_pydict( ... {"a": [[1.0, 0.0]], "b": [[0.0, 1.0]]} ... ) >>> result = df_orth.select( ... dfn.functions.cosine_distance( ... dfn.col("a"), dfn.col("b") ... ).alias("result") ... ) >>> result.collect_column("result")[0].as_py() 1.0 """ return Expr(f.cosine_distance(array1.expr, array2.expr)) def inner_product(array1: Expr, array2: Expr) -> Expr: """Returns the inner (dot) product of two numeric arrays. Treats each input as a vector and returns the sum of the element-wise products: ``sum(array1[i] * array2[i])``. For ``[1, 2, 3]`` and ``[4, 5, 6]`` the result is ``1*4 + 2*5 + 3*6 = 32``. Also available as :py:func:`dot_product` (and as ``dot_product`` in raw SQL). Both arrays must have the same length; otherwise execution fails. NULL is returned when either input array is NULL or when any element of either array is NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict( ... {"a": [[1.0, 2.0, 3.0]], "b": [[4.0, 5.0, 6.0]]} ... ) >>> result = df.select( ... dfn.functions.inner_product( ... dfn.col("a"), dfn.col("b") ... ).alias("result") ... ) >>> result.collect_column("result")[0].as_py() 32.0 NULL elements propagate to NULL output: >>> df_null = ctx.from_pydict( ... {"a": [[1.0, None, 3.0]], "b": [[4.0, 5.0, 6.0]]} ... ) >>> result = df_null.select( ... dfn.functions.inner_product( ... dfn.col("a"), dfn.col("b") ... ).alias("result") ... ) >>> result.collect_column("result")[0].as_py() is None True """ return Expr(f.inner_product(array1.expr, array2.expr)) def dot_product(array1: Expr, array2: Expr) -> Expr: """Returns the inner (dot) product of two numeric arrays. See Also: This is an alias for :py:func:`inner_product`. """ return inner_product(array1, array2) def list_cat(*args: Expr) -> Expr: """Concatenates the input arrays. See Also: This is an alias for :py:func:`array_concat`, :py:func:`array_cat`. """ return array_concat(*args) def list_concat(*args: Expr) -> Expr: """Concatenates the input arrays. See Also: This is an alias for :py:func:`array_concat`, :py:func:`array_cat`. """ return array_concat(*args) def list_distinct(array: Expr) -> Expr: """Returns distinct values from the array after removing duplicates. See Also: This is an alias for :py:func:`array_distinct`. """ return array_distinct(array) def list_compact(array: Expr) -> Expr: """Removes NULL values from the array. See Also: This is an alias for :py:func:`array_compact`. """ return array_compact(array) def list_normalize(array: Expr) -> Expr: """Scales a numeric array so it has Euclidean length 1. See Also: This is an alias for :py:func:`array_normalize`. """ return array_normalize(array) def list_dims(array: Expr) -> Expr: """Returns an array of the array's dimensions. See Also: This is an alias for :py:func:`array_dims`. """ return array_dims(array) def array_element(array: Expr, n: Expr | int) -> Expr: """Extracts the element with the index n from the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[10, 20, 30]]}) >>> result = df.select( ... dfn.functions.array_element(dfn.col("a"), 2).alias("result")) >>> result.collect_column("result")[0].as_py() 20 """ n = coerce_to_expr(n) return Expr(f.array_element(array.expr, n.expr)) def array_empty(array: Expr) -> Expr: """Returns a boolean indicating whether the array is empty. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2]]}) >>> result = df.select(dfn.functions.array_empty(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() False """ return Expr(f.array_empty(array.expr)) def list_empty(array: Expr) -> Expr: """Returns a boolean indicating whether the array is empty. See Also: This is an alias for :py:func:`array_empty`. """ return array_empty(array) def array_extract(array: Expr, n: Expr | int) -> Expr: """Extracts the element with the index n from the array. See Also: This is an alias for :py:func:`array_element`. """ return array_element(array, n) def list_element(array: Expr, n: Expr | int) -> Expr: """Extracts the element with the index n from the array. See Also: This is an alias for :py:func:`array_element`. """ return array_element(array, n) def list_extract(array: Expr, n: Expr | int) -> Expr: """Extracts the element with the index n from the array. See Also: This is an alias for :py:func:`array_element`. """ return array_element(array, n) def array_length(array: Expr) -> Expr: """Returns the length of the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select(dfn.functions.array_length(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() 3 """ return Expr(f.array_length(array.expr)) def list_length(array: Expr) -> Expr: """Returns the length of the array. See Also: This is an alias for :py:func:`array_length`. """ return array_length(array) def array_has(first_array: Expr, second_array: Expr) -> Expr: """Returns true if the element appears in the first array, otherwise false. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_has(dfn.col("a"), dfn.lit(2)).alias("result")) >>> result.collect_column("result")[0].as_py() True """ return Expr(f.array_has(first_array.expr, second_array.expr)) def array_has_all(first_array: Expr, second_array: Expr) -> Expr: """Determines if there is complete overlap ``second_array`` in ``first_array``. Returns true if each element of the second array appears in the first array. Otherwise, it returns false. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]], "b": [[1, 2]]}) >>> result = df.select( ... dfn.functions.array_has_all(dfn.col("a"), dfn.col("b")).alias("result")) >>> result.collect_column("result")[0].as_py() True """ return Expr(f.array_has_all(first_array.expr, second_array.expr)) def array_has_any(first_array: Expr, second_array: Expr) -> Expr: """Determine if there is an overlap between ``first_array`` and ``second_array``. Returns true if at least one element of the second array appears in the first array. Otherwise, it returns false. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]], "b": [[2, 5]]}) >>> result = df.select( ... dfn.functions.array_has_any(dfn.col("a"), dfn.col("b")).alias("result")) >>> result.collect_column("result")[0].as_py() True """ return Expr(f.array_has_any(first_array.expr, second_array.expr)) def array_contains(array: Expr, element: Expr) -> Expr: """Returns true if the element appears in the array, otherwise false. See Also: This is an alias for :py:func:`array_has`. """ return array_has(array, element) def list_has(array: Expr, element: Expr) -> Expr: """Returns true if the element appears in the array, otherwise false. See Also: This is an alias for :py:func:`array_has`. """ return array_has(array, element) def list_has_all(first_array: Expr, second_array: Expr) -> Expr: """Determines if there is complete overlap ``second_array`` in ``first_array``. See Also: This is an alias for :py:func:`array_has_all`. """ return array_has_all(first_array, second_array) def list_has_any(first_array: Expr, second_array: Expr) -> Expr: """Determine if there is an overlap between ``first_array`` and ``second_array``. See Also: This is an alias for :py:func:`array_has_any`. """ return array_has_any(first_array, second_array) def arrays_overlap(first_array: Expr, second_array: Expr) -> Expr: """Returns true if any element appears in both arrays. See Also: This is an alias for :py:func:`array_has_any`. """ return array_has_any(first_array, second_array) def list_overlap(first_array: Expr, second_array: Expr) -> Expr: """Returns true if any element appears in both arrays. See Also: This is an alias for :py:func:`array_has_any`. """ return array_has_any(first_array, second_array) def list_contains(array: Expr, element: Expr) -> Expr: """Returns true if the element appears in the array, otherwise false. See Also: This is an alias for :py:func:`array_has`. """ return array_has(array, element) def array_position(array: Expr, element: Expr, index: int | None = 1) -> Expr: """Return the position of the first occurrence of ``element`` in ``array``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[10, 20, 30]]}) >>> result = df.select( ... dfn.functions.array_position( ... dfn.col("a"), dfn.lit(20) ... ).alias("result")) >>> result.collect_column("result")[0].as_py() 2 Use ``index`` to start searching from a given position: >>> df = ctx.from_pydict({"a": [[10, 20, 10, 20]]}) >>> result = df.select( ... dfn.functions.array_position( ... dfn.col("a"), dfn.lit(20), index=3, ... ).alias("result")) >>> result.collect_column("result")[0].as_py() 4 """ return Expr(f.array_position(array.expr, element.expr, index)) def array_indexof(array: Expr, element: Expr, index: int | None = 1) -> Expr: """Return the position of the first occurrence of ``element`` in ``array``. See Also: This is an alias for :py:func:`array_position`. """ return array_position(array, element, index) def list_position(array: Expr, element: Expr, index: int | None = 1) -> Expr: """Return the position of the first occurrence of ``element`` in ``array``. See Also: This is an alias for :py:func:`array_position`. """ return array_position(array, element, index) def list_indexof(array: Expr, element: Expr, index: int | None = 1) -> Expr: """Return the position of the first occurrence of ``element`` in ``array``. See Also: This is an alias for :py:func:`array_position`. """ return array_position(array, element, index) def array_positions(array: Expr, element: Expr) -> Expr: """Searches for an element in the array and returns all occurrences. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1]]}) >>> result = df.select( ... dfn.functions.array_positions(dfn.col("a"), dfn.lit(1)).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 3] """ return Expr(f.array_positions(array.expr, element.expr)) def list_positions(array: Expr, element: Expr) -> Expr: """Searches for an element in the array and returns all occurrences. See Also: This is an alias for :py:func:`array_positions`. """ return array_positions(array, element) def array_ndims(array: Expr) -> Expr: """Returns the number of dimensions of the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select(dfn.functions.array_ndims(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() 1 """ return Expr(f.array_ndims(array.expr)) def list_ndims(array: Expr) -> Expr: """Returns the number of dimensions of the array. See Also: This is an alias for :py:func:`array_ndims`. """ return array_ndims(array) def array_prepend(element: Expr, array: Expr) -> Expr: """Prepends an element to the beginning of an array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2]]}) >>> result = df.select( ... dfn.functions.array_prepend(dfn.lit(0), dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() [0, 1, 2] """ return Expr(f.array_prepend(element.expr, array.expr)) def array_push_front(element: Expr, array: Expr) -> Expr: """Prepends an element to the beginning of an array. See Also: This is an alias for :py:func:`array_prepend`. """ return array_prepend(element, array) def list_prepend(element: Expr, array: Expr) -> Expr: """Prepends an element to the beginning of an array. See Also: This is an alias for :py:func:`array_prepend`. """ return array_prepend(element, array) def list_push_front(element: Expr, array: Expr) -> Expr: """Prepends an element to the beginning of an array. See Also: This is an alias for :py:func:`array_prepend`. """ return array_prepend(element, array) def array_pop_back(array: Expr) -> Expr: """Returns the array without the last element. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_pop_back(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2] """ return Expr(f.array_pop_back(array.expr)) def array_pop_front(array: Expr) -> Expr: """Returns the array without the first element. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_pop_front(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() [2, 3] """ return Expr(f.array_pop_front(array.expr)) def list_pop_back(array: Expr) -> Expr: """Returns the array without the last element. See Also: This is an alias for :py:func:`array_pop_back`. """ return array_pop_back(array) def list_pop_front(array: Expr) -> Expr: """Returns the array without the first element. See Also: This is an alias for :py:func:`array_pop_front`. """ return array_pop_front(array) def array_remove(array: Expr, element: Expr) -> Expr: """Removes the first element from the array equal to the given value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1]]}) >>> result = df.select( ... dfn.functions.array_remove(dfn.col("a"), dfn.lit(1)).alias("result")) >>> result.collect_column("result")[0].as_py() [2, 1] """ return Expr(f.array_remove(array.expr, element.expr)) def list_remove(array: Expr, element: Expr) -> Expr: """Removes the first element from the array equal to the given value. See Also: This is an alias for :py:func:`array_remove`. """ return array_remove(array, element) def array_remove_n(array: Expr, element: Expr, max: Expr | int) -> Expr: """Removes the first ``max`` elements from the array equal to the given value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1, 1]]}) >>> result = df.select( ... dfn.functions.array_remove_n( ... dfn.col("a"), dfn.lit(1), 2 ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [2, 1] """ max = coerce_to_expr(max) return Expr(f.array_remove_n(array.expr, element.expr, max.expr)) def list_remove_n(array: Expr, element: Expr, max: Expr | int) -> Expr: """Removes the first ``max`` elements from the array equal to the given value. See Also: This is an alias for :py:func:`array_remove_n`. """ return array_remove_n(array, element, max) def array_remove_all(array: Expr, element: Expr) -> Expr: """Removes all elements from the array equal to the given value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1]]}) >>> result = df.select( ... dfn.functions.array_remove_all( ... dfn.col("a"), dfn.lit(1) ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [2] """ return Expr(f.array_remove_all(array.expr, element.expr)) def list_remove_all(array: Expr, element: Expr) -> Expr: """Removes all elements from the array equal to the given value. See Also: This is an alias for :py:func:`array_remove_all`. """ return array_remove_all(array, element) def array_repeat(element: Expr, count: Expr | int) -> Expr: """Returns an array containing ``element`` ``count`` times. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.array_repeat(dfn.lit(3), 3).alias("result")) >>> result.collect_column("result")[0].as_py() [3, 3, 3] """ count = coerce_to_expr(count) return Expr(f.array_repeat(element.expr, count.expr)) def list_repeat(element: Expr, count: Expr | int) -> Expr: """Returns an array containing ``element`` ``count`` times. See Also: This is an alias for :py:func:`array_repeat`. """ return array_repeat(element, count) def array_replace(array: Expr, from_val: Expr, to_val: Expr) -> Expr: """Replaces the first occurrence of ``from_val`` with ``to_val``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1]]}) >>> result = df.select( ... dfn.functions.array_replace(dfn.col("a"), dfn.lit(1), ... dfn.lit(9)).alias("result")) >>> result.collect_column("result")[0].as_py() [9, 2, 1] """ return Expr(f.array_replace(array.expr, from_val.expr, to_val.expr)) def list_replace(array: Expr, from_val: Expr, to_val: Expr) -> Expr: """Replaces the first occurrence of ``from_val`` with ``to_val``. See Also: This is an alias for :py:func:`array_replace`. """ return array_replace(array, from_val, to_val) def array_replace_n(array: Expr, from_val: Expr, to_val: Expr, max: Expr | int) -> Expr: """Replace ``n`` occurrences of ``from_val`` with ``to_val``. Replaces the first ``max`` occurrences of the specified element with another specified element. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1, 1]]}) >>> result = df.select( ... dfn.functions.array_replace_n( ... dfn.col("a"), dfn.lit(1), dfn.lit(9), 2 ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [9, 2, 9, 1] """ max = coerce_to_expr(max) return Expr(f.array_replace_n(array.expr, from_val.expr, to_val.expr, max.expr)) def list_replace_n(array: Expr, from_val: Expr, to_val: Expr, max: Expr | int) -> Expr: """Replace ``n`` occurrences of ``from_val`` with ``to_val``. Replaces the first ``max`` occurrences of the specified element with another specified element. See Also: This is an alias for :py:func:`array_replace_n`. """ return array_replace_n(array, from_val, to_val, max) def array_replace_all(array: Expr, from_val: Expr, to_val: Expr) -> Expr: """Replaces all occurrences of ``from_val`` with ``to_val``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 1]]}) >>> result = df.select( ... dfn.functions.array_replace_all(dfn.col("a"), dfn.lit(1), ... dfn.lit(9)).alias("result")) >>> result.collect_column("result")[0].as_py() [9, 2, 9] """ return Expr(f.array_replace_all(array.expr, from_val.expr, to_val.expr)) def list_replace_all(array: Expr, from_val: Expr, to_val: Expr) -> Expr: """Replaces all occurrences of ``from_val`` with ``to_val``. See Also: This is an alias for :py:func:`array_replace_all`. """ return array_replace_all(array, from_val, to_val) def array_sort(array: Expr, descending: bool = False, null_first: bool = False) -> Expr: """Sort an array. Args: array: The input array to sort. descending: If True, sorts in descending order. null_first: If True, nulls will be returned at the beginning of the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[3, 1, 2]]}) >>> result = df.select( ... dfn.functions.array_sort( ... dfn.col("a") ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2, 3] >>> df = ctx.from_pydict({"a": [[3, None, 1]]}) >>> result = df.select( ... dfn.functions.array_sort( ... dfn.col("a"), descending=True, null_first=True, ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [None, 3, 1] """ desc = "DESC" if descending else "ASC" nulls_first = "NULLS FIRST" if null_first else "NULLS LAST" return Expr( f.array_sort( array.expr, Expr.literal(pa.scalar(desc, type=pa.string())).expr, Expr.literal(pa.scalar(nulls_first, type=pa.string())).expr, ) ) def list_sort(array: Expr, descending: bool = False, null_first: bool = False) -> Expr: """Sorts the array. See Also: This is an alias for :py:func:`array_sort`. """ return array_sort(array, descending=descending, null_first=null_first) def array_slice( array: Expr, begin: Expr | int, end: Expr | int, stride: Expr | int | None = None, ) -> Expr: """Returns a slice of the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3, 4]]}) >>> result = df.select( ... dfn.functions.array_slice(dfn.col("a"), 2, 3).alias("result")) >>> result.collect_column("result")[0].as_py() [2, 3] Use ``stride`` to skip elements: >>> result = df.select( ... dfn.functions.array_slice( ... dfn.col("a"), 1, 4, stride=2, ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 3] """ begin = coerce_to_expr(begin) end = coerce_to_expr(end) stride = coerce_to_expr_or_none(stride) return Expr( f.array_slice( array.expr, begin.expr, end.expr, stride.expr if stride is not None else None, ) ) def list_slice( array: Expr, begin: Expr | int, end: Expr | int, stride: Expr | int | None = None ) -> Expr: """Returns a slice of the array. See Also: This is an alias for :py:func:`array_slice`. """ return array_slice(array, begin, end, stride) def array_intersect(array1: Expr, array2: Expr) -> Expr: """Returns the intersection of ``array1`` and ``array2``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]], "b": [[2, 3, 4]]}) >>> result = df.select( ... dfn.functions.array_intersect( ... dfn.col("a"), dfn.col("b") ... ).alias("result") ... ) >>> sorted( ... result.collect_column("result")[0].as_py() ... ) [2, 3] """ return Expr(f.array_intersect(array1.expr, array2.expr)) def list_intersect(array1: Expr, array2: Expr) -> Expr: """Returns an the intersection of ``array1`` and ``array2``. See Also: This is an alias for :py:func:`array_intersect`. """ return array_intersect(array1, array2) def array_union(array1: Expr, array2: Expr) -> Expr: """Returns an array of the elements in the union of array1 and array2. Duplicate rows will not be returned. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]], "b": [[2, 3, 4]]}) >>> result = df.select( ... dfn.functions.array_union( ... dfn.col("a"), dfn.col("b") ... ).alias("result") ... ) >>> sorted( ... result.collect_column("result")[0].as_py() ... ) [1, 2, 3, 4] """ return Expr(f.array_union(array1.expr, array2.expr)) def list_union(array1: Expr, array2: Expr) -> Expr: """Returns an array of the elements in the union of array1 and array2. Duplicate rows will not be returned. See Also: This is an alias for :py:func:`array_union`. """ return array_union(array1, array2) def array_except(array1: Expr, array2: Expr) -> Expr: """Returns the elements that appear in ``array1`` but not in ``array2``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]], "b": [[2, 3, 4]]}) >>> result = df.select( ... dfn.functions.array_except(dfn.col("a"), dfn.col("b")).alias("result")) >>> result.collect_column("result")[0].as_py() [1] """ return Expr(f.array_except(array1.expr, array2.expr)) def list_except(array1: Expr, array2: Expr) -> Expr: """Returns the elements that appear in ``array1`` but not in the ``array2``. See Also: This is an alias for :py:func:`array_except`. """ return array_except(array1, array2) def array_resize(array: Expr, size: Expr | int, value: Expr) -> Expr: """Returns an array with the specified size filled. If ``size`` is greater than the ``array`` length, the additional entries will be filled with the given ``value``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2]]}) >>> result = df.select( ... dfn.functions.array_resize(dfn.col("a"), 4, dfn.lit(0)).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2, 0, 0] """ size = coerce_to_expr(size) return Expr(f.array_resize(array.expr, size.expr, value.expr)) def list_resize(array: Expr, size: Expr | int, value: Expr) -> Expr: """Returns an array with the specified size filled. If ``size`` is greater than the ``array`` length, the additional entries will be filled with the given ``value``. See Also: This is an alias for :py:func:`array_resize`. """ return array_resize(array, size, value) def array_any_value(array: Expr) -> Expr: """Returns the first non-null element in the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[None, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_any_value(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() 2 """ return Expr(f.array_any_value(array.expr)) def list_any_value(array: Expr) -> Expr: """Returns the first non-null element in the array. See Also: This is an alias for :py:func:`array_any_value`. """ return array_any_value(array) def array_distance(array1: Expr, array2: Expr) -> Expr: """Returns the Euclidean distance between two numeric arrays. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1.0, 2.0]], "b": [[1.0, 4.0]]}) >>> result = df.select( ... dfn.functions.array_distance( ... dfn.col("a"), dfn.col("b"), ... ).alias("result")) >>> result.collect_column("result")[0].as_py() 2.0 """ return Expr(f.array_distance(array1.expr, array2.expr)) def list_distance(array1: Expr, array2: Expr) -> Expr: """Returns the Euclidean distance between two numeric arrays. See Also: This is an alias for :py:func:`array_distance`. """ return array_distance(array1, array2) def array_max(array: Expr) -> Expr: """Returns the maximum value in the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_max(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() 3 """ return Expr(f.array_max(array.expr)) def list_max(array: Expr) -> Expr: """Returns the maximum value in the array. See Also: This is an alias for :py:func:`array_max`. """ return array_max(array) def array_min(array: Expr) -> Expr: """Returns the minimum value in the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_min(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() 1 """ return Expr(f.array_min(array.expr)) def list_min(array: Expr) -> Expr: """Returns the minimum value in the array. See Also: This is an alias for :py:func:`array_min`. """ return array_min(array) def array_reverse(array: Expr) -> Expr: """Reverses the order of elements in the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select( ... dfn.functions.array_reverse(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() [3, 2, 1] """ return Expr(f.array_reverse(array.expr)) def list_reverse(array: Expr) -> Expr: """Reverses the order of elements in the array. See Also: This is an alias for :py:func:`array_reverse`. """ return array_reverse(array) def arrays_zip(*arrays: Expr) -> Expr: """Combines multiple arrays into a single array of structs. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2]], "b": [[3, 4]]}) >>> result = df.select( ... dfn.functions.arrays_zip(dfn.col("a"), dfn.col("b")).alias("result")) >>> result.collect_column("result")[0].as_py() [{'1': 1, '2': 3}, {'1': 2, '2': 4}] """ args = [a.expr for a in arrays] return Expr(f.arrays_zip(args)) def list_zip(*arrays: Expr) -> Expr: """Combines multiple arrays into a single array of structs. See Also: This is an alias for :py:func:`arrays_zip`. """ return arrays_zip(*arrays) def string_to_array( string: Expr, delimiter: Expr | str, null_string: Expr | str | None = None ) -> Expr: """Splits a string based on a delimiter and returns an array of parts. Any parts matching the optional ``null_string`` will be replaced with ``NULL``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["hello,world"]}) >>> result = df.select( ... dfn.functions.string_to_array(dfn.col("a"), ",").alias("result")) >>> result.collect_column("result")[0].as_py() ['hello', 'world'] Replace parts matching a ``null_string`` with ``NULL``: >>> result = df.select( ... dfn.functions.string_to_array( ... dfn.col("a"), ",", null_string="world", ... ).alias("result")) >>> result.collect_column("result")[0].as_py() ['hello', None] """ delimiter = coerce_to_expr(delimiter) null_string = coerce_to_expr_or_none(null_string) return Expr( f.string_to_array( string.expr, delimiter.expr, null_string.expr if null_string is not None else None, ) ) def string_to_list( string: Expr, delimiter: Expr | str, null_string: Expr | str | None = None ) -> Expr: """Splits a string based on a delimiter and returns an array of parts. See Also: This is an alias for :py:func:`string_to_array`. """ return string_to_array(string, delimiter, null_string) def gen_series(start: Expr, stop: Expr, step: Expr | None = None) -> Expr: """Creates a list of values in the range between start and stop. Unlike :py:func:`range`, this includes the upper bound. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0]}) >>> result = df.select( ... dfn.functions.gen_series( ... dfn.lit(1), dfn.lit(5), ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2, 3, 4, 5] Specify a custom ``step``: >>> result = df.select( ... dfn.functions.gen_series( ... dfn.lit(1), dfn.lit(10), step=dfn.lit(3), ... ).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 4, 7, 10] """ step_expr = step.expr if step is not None else None return Expr(f.gen_series(start.expr, stop.expr, step_expr)) def generate_series(start: Expr, stop: Expr, step: Expr | None = None) -> Expr: """Creates a list of values in the range between start and stop. Unlike :py:func:`range`, this includes the upper bound. See Also: This is an alias for :py:func:`gen_series`. """ return gen_series(start, stop, step) def flatten(array: Expr) -> Expr: """Flattens an array of arrays into a single array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[[1, 2], [3, 4]]]}) >>> result = df.select(dfn.functions.flatten(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() [1, 2, 3, 4] """ return Expr(f.flatten(array.expr)) def cardinality(array: Expr) -> Expr: """Returns the total number of elements in the array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [[1, 2, 3]]}) >>> result = df.select(dfn.functions.cardinality(dfn.col("a")).alias("result")) >>> result.collect_column("result")[0].as_py() 3 """ return Expr(f.cardinality(array.expr)) def empty(array: Expr) -> Expr: """Returns true if the array is empty. See Also: This is an alias for :py:func:`array_empty`. """ return array_empty(array) # map functions def make_map(*args: Any) -> Expr: """Returns a map expression. Supports three calling conventions: - ``make_map({"a": 1, "b": 2})`` — from a Python dictionary. - ``make_map([keys], [values])`` — from a list of keys and a list of their associated values. Both lists must be the same length. - ``make_map(k1, v1, k2, v2, ...)`` — from alternating keys and their associated values. Keys and values that are not already :py:class:`~datafusion.expr.Expr` are automatically converted to literal expressions. Examples: From a dictionary: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.make_map({"a": 1, "b": 2}).alias("m")) >>> result.collect_column("m")[0].as_py() [('a', 1), ('b', 2)] From two lists: >>> df = ctx.from_pydict({"key": ["x", "y"], "val": [10, 20]}) >>> df = df.select( ... dfn.functions.make_map( ... [dfn.col("key")], [dfn.col("val")] ... ).alias("m")) >>> df.collect_column("m")[0].as_py() [('x', 10)] From alternating keys and values: >>> df = ctx.from_pydict({"a": [1]}) >>> result = df.select( ... dfn.functions.make_map("x", 1, "y", 2).alias("m")) >>> result.collect_column("m")[0].as_py() [('x', 1), ('y', 2)] """ if len(args) == 1 and isinstance(args[0], dict): key_list = list(args[0].keys()) value_list = list(args[0].values()) elif ( len(args) == 2 # noqa: PLR2004 and isinstance(args[0], list) and isinstance(args[1], list) ): if len(args[0]) != len(args[1]): msg = "make_map requires key and value lists to be the same length" raise ValueError(msg) key_list = args[0] value_list = args[1] elif len(args) >= 2 and len(args) % 2 == 0: # noqa: PLR2004 key_list = list(args[0::2]) value_list = list(args[1::2]) else: msg = ( "make_map expects a dict, two lists, or an even number of " "key-value arguments" ) raise ValueError(msg) key_exprs = [k if isinstance(k, Expr) else Expr.literal(k) for k in key_list] val_exprs = [v if isinstance(v, Expr) else Expr.literal(v) for v in value_list] return Expr(f.make_map([k.expr for k in key_exprs], [v.expr for v in val_exprs])) def map_keys(map: Expr) -> Expr: """Returns a list of all keys in the map. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> df = df.select( ... dfn.functions.make_map({"x": 1, "y": 2}).alias("m")) >>> result = df.select( ... dfn.functions.map_keys(dfn.col("m")).alias("keys")) >>> result.collect_column("keys")[0].as_py() ['x', 'y'] """ return Expr(f.map_keys(map.expr)) def map_values(map: Expr) -> Expr: """Returns a list of all values in the map. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> df = df.select( ... dfn.functions.make_map({"x": 1, "y": 2}).alias("m")) >>> result = df.select( ... dfn.functions.map_values(dfn.col("m")).alias("vals")) >>> result.collect_column("vals")[0].as_py() [1, 2] """ return Expr(f.map_values(map.expr)) def map_extract(map: Expr, key: Expr) -> Expr: """Returns the value for a given key in the map. Returns ``[None]`` if the key is absent. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> df = df.select( ... dfn.functions.make_map({"x": 1, "y": 2}).alias("m")) >>> result = df.select( ... dfn.functions.map_extract( ... dfn.col("m"), dfn.lit("x") ... ).alias("val")) >>> result.collect_column("val")[0].as_py() [1] """ return Expr(f.map_extract(map.expr, key.expr)) def map_entries(map: Expr) -> Expr: """Returns a list of all entries (key-value struct pairs) in the map. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1]}) >>> df = df.select( ... dfn.functions.make_map({"x": 1, "y": 2}).alias("m")) >>> result = df.select( ... dfn.functions.map_entries(dfn.col("m")).alias("entries")) >>> result.collect_column("entries")[0].as_py() [{'key': 'x', 'value': 1}, {'key': 'y', 'value': 2}] """ return Expr(f.map_entries(map.expr)) def element_at(map: Expr, key: Expr) -> Expr: """Returns the value for a given key in the map. Returns ``[None]`` if the key is absent. See Also: This is an alias for :py:func:`map_extract`. """ return map_extract(map, key) # aggregate functions def approx_distinct( expression: Expr, filter: Expr | None = None, ) -> Expr: """Returns the approximate number of distinct values. This aggregate function is similar to :py:func:`count` with distinct set, but it will approximate the number of distinct entries. It may return significantly faster than :py:func:`count` for some DataFrames. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: Values to check for distinct entries filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.approx_distinct( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() == 3 True >>> result = df.aggregate( ... [], [dfn.functions.approx_distinct( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() == 2 True """ filter_raw = filter.expr if filter is not None else None return Expr(f.approx_distinct(expression.expr, filter=filter_raw)) def approx_median(expression: Expr, filter: Expr | None = None) -> Expr: """Returns the approximate median value. This aggregate function is similar to :py:func:`median`, but it will only approximate the median. It may return significantly faster for some DataFrames. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by`` and ``null_treatment``, and ``distinct``. Args: expression: Values to find the median for filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.approx_median( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.approx_median( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.approx_median(expression.expr, filter=filter_raw)) def approx_percentile_cont( sort_expression: Expr | SortExpr, percentile: float, num_centroids: int | None = None, filter: Expr | None = None, ) -> Expr: """Returns the value that is approximately at a given percentile of ``expr``. This aggregate function assumes the input values form a continuous distribution. Suppose you have a DataFrame which consists of 100 different test scores. If you called this function with a percentile of 0.9, it would return the value of the test score that is above 90% of the other test scores. The returned value may be between two of the values. This function uses the [t-digest](https://arxiv.org/abs/1902.04023) algorithm to compute the percentile. You can limit the number of bins used in this algorithm by setting the ``num_centroids`` parameter. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: sort_expression: Values for which to find the approximate percentile percentile: This must be between 0.0 and 1.0, inclusive num_centroids: Max bin size for the t-digest algorithm filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0, 4.0, 5.0]}) >>> result = df.aggregate( ... [], [dfn.functions.approx_percentile_cont( ... dfn.col("a"), 0.5 ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3.0 >>> result = df.aggregate( ... [], [dfn.functions.approx_percentile_cont( ... dfn.col("a"), 0.5, ... num_centroids=10, ... filter=dfn.col("a") > dfn.lit(1.0), ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3.5 """ sort_expr_raw = sort_or_default(sort_expression) filter_raw = filter.expr if filter is not None else None return Expr( f.approx_percentile_cont( sort_expr_raw, percentile, num_centroids=num_centroids, filter=filter_raw ) ) def approx_percentile_cont_with_weight( sort_expression: Expr | SortExpr, weight: Expr, percentile: float, num_centroids: int | None = None, filter: Expr | None = None, ) -> Expr: """Returns the value of the weighted approximate percentile. This aggregate function is similar to :py:func:`approx_percentile_cont` except that it uses the associated associated weights. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: sort_expression: Values for which to find the approximate percentile weight: Relative weight for each of the values in ``expression`` percentile: This must be between 0.0 and 1.0, inclusive num_centroids: Max bin size for the t-digest algorithm filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0], "w": [1.0, 1.0, 1.0]}) >>> result = df.aggregate( ... [], [dfn.functions.approx_percentile_cont_with_weight( ... dfn.col("a"), dfn.col("w"), 0.5 ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.approx_percentile_cont_with_weight( ... dfn.col("a"), dfn.col("w"), 0.5, ... num_centroids=10, ... filter=dfn.col("a") > dfn.lit(1.0), ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.5 """ sort_expr_raw = sort_or_default(sort_expression) filter_raw = filter.expr if filter is not None else None return Expr( f.approx_percentile_cont_with_weight( sort_expr_raw, weight.expr, percentile, num_centroids=num_centroids, filter=filter_raw, ) ) def percentile_cont( sort_expression: Expr | SortExpr, percentile: float, filter: Expr | None = None, ) -> Expr: """Computes the exact percentile of input values using continuous interpolation. Unlike :py:func:`approx_percentile_cont`, this function computes the exact percentile value rather than an approximation. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: sort_expression: Values for which to find the percentile percentile: This must be between 0.0 and 1.0, inclusive filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0, 4.0, 5.0]}) >>> result = df.aggregate( ... [], [dfn.functions.percentile_cont( ... dfn.col("a"), 0.5 ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3.0 >>> result = df.aggregate( ... [], [dfn.functions.percentile_cont( ... dfn.col("a"), 0.5, ... filter=dfn.col("a") > dfn.lit(1.0), ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3.5 """ sort_expr_raw = sort_or_default(sort_expression) filter_raw = filter.expr if filter is not None else None return Expr(f.percentile_cont(sort_expr_raw, percentile, filter=filter_raw)) def quantile_cont( sort_expression: Expr | SortExpr, percentile: float, filter: Expr | None = None, ) -> Expr: """Computes the exact percentile of input values using continuous interpolation. See Also: This is an alias for :py:func:`percentile_cont`. """ return percentile_cont(sort_expression, percentile, filter) def array_agg( expression: Expr, distinct: bool = False, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Aggregate values into an array. Currently ``distinct`` and ``order_by`` cannot be used together. As a work around, consider :py:func:`array_sort` after aggregation. [Issue Tracker](https://github.com/apache/datafusion/issues/12371) If using the builder functions described in ref:`_aggregation` this function ignores the option ``null_treatment``. Args: expression: Values to combine into an array distinct: If True, a single entry for each distinct value will be in the result filter: If provided, only compute against rows for which the filter is True order_by: Order the resultant array values. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.array_agg( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() [1, 2, 3] >>> df = ctx.from_pydict({"a": [3, 1, 2, 1]}) >>> result = df.aggregate( ... [], [dfn.functions.array_agg( ... dfn.col("a"), distinct=True, ... ).alias("v")]) >>> sorted(result.collect_column("v")[0].as_py()) [1, 2, 3] >>> result = df.aggregate( ... [], [dfn.functions.array_agg( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1), ... order_by="a", ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() [2, 3] """ order_by_raw = sort_list_to_raw_sort_list(order_by) filter_raw = filter.expr if filter is not None else None return Expr( f.array_agg( expression.expr, distinct=distinct, filter=filter_raw, order_by=order_by_raw ) ) def grouping( expression: Expr, distinct: bool = False, filter: Expr | None = None, ) -> Expr: """Indicates whether a column is aggregated across in the current row. Returns 0 when the column is part of the grouping key for that row (i.e., the row contains per-group results for that column). Returns 1 when the column is *not* part of the grouping key (i.e., the row's aggregate spans all values of that column). This function is meaningful with :py:meth:`GroupingSet.rollup `, :py:meth:`GroupingSet.cube `, or :py:meth:`GroupingSet.grouping_sets `, where different rows are grouped by different subsets of columns. In a default aggregation without grouping sets every column is always part of the key, so ``grouping()`` always returns 0. .. warning:: Due to an upstream DataFusion limitation (`#21411 `_), ``.alias()`` cannot be applied directly to a ``grouping()`` expression. Doing so will raise an error at execution time. To rename the column, use :py:meth:`~datafusion.dataframe.DataFrame.with_column_renamed` on the result DataFrame instead. Args: expression: The column to check grouping status for distinct: If True, compute on distinct values only filter: If provided, only compute against rows for which the filter is True Examples: With :py:meth:`~datafusion.expr.GroupingSet.rollup`, the result includes both per-group rows (``grouping(a) = 0``) and a grand-total row where ``a`` is aggregated across (``grouping(a) = 1``): >>> from datafusion.expr import GroupingSet >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 1, 2], "b": [10, 20, 30]}) >>> result = df.aggregate( ... [GroupingSet.rollup(dfn.col("a"))], ... [dfn.functions.sum(dfn.col("b")).alias("s"), ... dfn.functions.grouping(dfn.col("a"))], ... ).sort(dfn.col("a").sort(nulls_first=False)) >>> result.collect_column("s").to_pylist() [30, 30, 60] See Also: :py:class:`~datafusion.expr.GroupingSet` """ filter_raw = filter.expr if filter is not None else None return Expr(f.grouping(expression.expr, distinct=distinct, filter=filter_raw)) def avg( expression: Expr, distinct: bool = False, filter: Expr | None = None, ) -> Expr: """Returns the average value. This aggregate function expects a numeric expression and will return a float. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by`` and ``null_treatment``. Args: expression: Values to combine into an array distinct: If True, duplicate values are removed before averaging. filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.avg( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.avg( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.5 >>> df = ctx.from_pydict({"a": [1.0, 1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.avg( ... dfn.col("a"), distinct=True, ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 """ filter_raw = filter.expr if filter is not None else None return Expr(f.avg(expression.expr, distinct=distinct, filter=filter_raw)) def corr(value_y: Expr, value_x: Expr, filter: Expr | None = None) -> Expr: """Returns the correlation coefficient between ``value1`` and ``value2``. This aggregate function expects both values to be numeric and will return a float. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: value_y: The dependent variable for correlation value_x: The independent variable for correlation filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0], "b": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.corr( ... dfn.col("a"), dfn.col("b") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.0 >>> result = df.aggregate( ... [], [dfn.functions.corr( ... dfn.col("a"), dfn.col("b"), ... filter=dfn.col("a") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.0 """ filter_raw = filter.expr if filter is not None else None return Expr(f.corr(value_y.expr, value_x.expr, filter=filter_raw)) def count( expressions: Expr | list[Expr] | None = None, distinct: bool = False, filter: Expr | None = None, ) -> Expr: """Returns the number of rows that match the given arguments. This aggregate function will count the non-null rows provided in the expression. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by`` and ``null_treatment``. Args: expressions: Argument to perform bitwise calculation on distinct: If True, a single entry for each distinct value will be in the result filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.count( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3 >>> df = ctx.from_pydict({"a": [1, 1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.count( ... dfn.col("a"), distinct=True, ... filter=dfn.col("a") > dfn.lit(1), ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2 """ filter_raw = filter.expr if filter is not None else None if expressions is None: args = [Expr.literal(1).expr] elif isinstance(expressions, list): args = [arg.expr for arg in expressions] else: args = [expressions.expr] return Expr(f.count(*args, distinct=distinct, filter=filter_raw)) def covar_pop(value_y: Expr, value_x: Expr, filter: Expr | None = None) -> Expr: """Computes the population covariance. This aggregate function expects both values to be numeric and will return a float. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: value_y: The dependent variable for covariance value_x: The independent variable for covariance filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 5.0, 10.0], "b": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], ... [dfn.functions.covar_pop( ... dfn.col("a"), dfn.col("b") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 3.0 >>> df = ctx.from_pydict( ... {"a": [0.0, 1.0, 3.0], "b": [0.0, 1.0, 3.0]}) >>> result = df.aggregate( ... [], ... [dfn.functions.covar_pop( ... dfn.col("a"), dfn.col("b"), ... filter=dfn.col("a") > dfn.lit(0.0) ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 1.0 """ filter_raw = filter.expr if filter is not None else None return Expr(f.covar_pop(value_y.expr, value_x.expr, filter=filter_raw)) def covar_samp(value_y: Expr, value_x: Expr, filter: Expr | None = None) -> Expr: """Computes the sample covariance. This aggregate function expects both values to be numeric and will return a float. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: value_y: The dependent variable for covariance value_x: The independent variable for covariance filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0], "b": [4.0, 5.0, 6.0]}) >>> result = df.aggregate( ... [], [dfn.functions.covar_samp( ... dfn.col("a"), dfn.col("b") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.0 >>> result = df.aggregate( ... [], [dfn.functions.covar_samp( ... dfn.col("a"), dfn.col("b"), ... filter=dfn.col("a") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.covar_samp(value_y.expr, value_x.expr, filter=filter_raw)) def covar(value_y: Expr, value_x: Expr, filter: Expr | None = None) -> Expr: """Computes the sample covariance. See Also: This is an alias for :py:func:`covar_samp`. """ return covar_samp(value_y, value_x, filter) def max(expression: Expr, filter: Expr | None = None) -> Expr: """Aggregate function that returns the maximum value of the argument. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: The value to find the maximum of filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.max( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3 >>> result = df.aggregate( ... [], [dfn.functions.max( ... dfn.col("a"), ... filter=dfn.col("a") < dfn.lit(3) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2 """ filter_raw = filter.expr if filter is not None else None return Expr(f.max(expression.expr, filter=filter_raw)) def mean(expression: Expr, filter: Expr | None = None) -> Expr: """Returns the average (mean) value of the argument. See Also: This is an alias for :py:func:`avg`. """ return avg(expression, filter) def median( expression: Expr, distinct: bool = False, filter: Expr | None = None ) -> Expr: """Computes the median of a set of numbers. This aggregate function returns the median value of the expression for the given aggregate function. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by`` and ``null_treatment``. Args: expression: The value to compute the median of distinct: If True, a single entry for each distinct value will be in the result filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.median( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> df = ctx.from_pydict({"a": [1.0, 1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.median( ... dfn.col("a"), distinct=True, ... filter=dfn.col("a") < dfn.lit(3.0), ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.median(expression.expr, distinct=distinct, filter=filter_raw)) def min(expression: Expr, filter: Expr | None = None) -> Expr: """Aggregate function that returns the minimum value of the argument. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: The value to find the minimum of filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.min( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1 >>> result = df.aggregate( ... [], [dfn.functions.min( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2 """ filter_raw = filter.expr if filter is not None else None return Expr(f.min(expression.expr, filter=filter_raw)) def sum( expression: Expr, distinct: bool = False, filter: Expr | None = None, ) -> Expr: """Computes the sum of a set of numbers. This aggregate function expects a numeric expression. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by`` and ``null_treatment``. Args: expression: Values to combine into an array distinct: If True, duplicate values are removed before summing. filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.sum( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 6 >>> result = df.aggregate( ... [], [dfn.functions.sum( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 5 >>> df = ctx.from_pydict({"a": [1, 1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.sum( ... dfn.col("a"), distinct=True, ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 6 """ filter_raw = filter.expr if filter is not None else None return Expr(f.sum(expression.expr, distinct=distinct, filter=filter_raw)) def stddev(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the standard deviation of the argument. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: The value to find the minimum of filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [2.0, 4.0, 6.0]}) >>> result = df.aggregate( ... [], [dfn.functions.stddev( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.stddev( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(2.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.41... """ filter_raw = filter.expr if filter is not None else None return Expr(f.stddev(expression.expr, filter=filter_raw)) def stddev_pop(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the population standard deviation of the argument. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: The value to find the minimum of filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [0.0, 1.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.stddev_pop( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 1.247... >>> df = ctx.from_pydict({"a": [0.0, 1.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.stddev_pop( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(0.0) ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 1.0 """ filter_raw = filter.expr if filter is not None else None return Expr(f.stddev_pop(expression.expr, filter=filter_raw)) def stddev_samp(arg: Expr, filter: Expr | None = None) -> Expr: """Computes the sample standard deviation of the argument. See Also: This is an alias for :py:func:`stddev`. """ return stddev(arg, filter=filter) def var(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the sample variance of the argument. See Also: This is an alias for :py:func:`var_samp`. """ return var_samp(expression, filter) def var_pop(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the population variance of the argument. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: The variable to compute the variance for filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [-1.0, 0.0, 2.0]}) >>> result = df.aggregate( ... [], [dfn.functions.var_pop( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.555... >>> result = df.aggregate( ... [], [dfn.functions.var_pop( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(-1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.0 """ filter_raw = filter.expr if filter is not None else None return Expr(f.var_pop(expression.expr, filter=filter_raw)) def var_population(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the population variance of the argument. See Also: This is an alias for :py:func:`var_pop`. """ return var_pop(expression, filter) def var_samp(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the sample variance of the argument. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: The variable to compute the variance for filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.var_samp( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.0 >>> result = df.aggregate( ... [], [dfn.functions.var_samp( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.var_sample(expression.expr, filter=filter_raw)) def var_sample(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the sample variance of the argument. See Also: This is an alias for :py:func:`var_samp`. """ return var_samp(expression, filter) def regr_avgx( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the average of the independent variable ``x``. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [1.0, 2.0, 3.0], "x": [4.0, 5.0, 6.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_avgx( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 5.0 >>> result = df.aggregate( ... [], [dfn.functions.regr_avgx( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 5.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_avgx(y.expr, x.expr, filter=filter_raw)) def regr_avgy( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the average of the dependent variable ``y``. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [1.0, 2.0, 3.0], "x": [4.0, 5.0, 6.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_avgy( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.regr_avgy( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_avgy(y.expr, x.expr, filter=filter_raw)) def regr_count( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Counts the number of rows in which both expressions are not null. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [1.0, 2.0, 3.0], "x": [4.0, 5.0, 6.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_count( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3 >>> result = df.aggregate( ... [], [dfn.functions.regr_count( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_count(y.expr, x.expr, filter=filter_raw)) def regr_intercept( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the intercept from the linear regression. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [2.0, 4.0, 6.0], "x": [4.0, 16.0, 36.0]}) >>> result = df.aggregate( ... [], ... [dfn.functions.regr_intercept( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.714... >>> result = df.aggregate( ... [], ... [dfn.functions.regr_intercept( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(2.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.4 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_intercept(y.expr, x.expr, filter=filter_raw)) def regr_r2( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the R-squared value from linear regression. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [2.0, 4.0, 6.0], "x": [4.0, 16.0, 36.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_r2( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.9795... >>> result = df.aggregate( ... [], [dfn.functions.regr_r2( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(2.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 1.0 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_r2(y.expr, x.expr, filter=filter_raw)) def regr_slope( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the slope from linear regression. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [2.0, 4.0, 6.0], "x": [4.0, 16.0, 36.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_slope( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.122... >>> result = df.aggregate( ... [], [dfn.functions.regr_slope( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(2.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.1 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_slope(y.expr, x.expr, filter=filter_raw)) def regr_sxx( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the sum of squares of the independent variable ``x``. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [1.0, 2.0, 3.0], "x": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_sxx( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.regr_sxx( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_sxx(y.expr, x.expr, filter=filter_raw)) def regr_sxy( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the sum of products of pairs of numbers. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [1.0, 2.0, 3.0], "x": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_sxy( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.regr_sxy( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_sxy(y.expr, x.expr, filter=filter_raw)) def regr_syy( y: Expr, x: Expr, filter: Expr | None = None, ) -> Expr: """Computes the sum of squares of the dependent variable ``y``. This is a linear regression aggregate function. Only non-null pairs of the inputs are evaluated. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: y: The linear regression dependent variable x: The linear regression independent variable filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"y": [1.0, 2.0, 3.0], "x": [1.0, 2.0, 3.0]}) >>> result = df.aggregate( ... [], [dfn.functions.regr_syy( ... dfn.col("y"), dfn.col("x") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 2.0 >>> result = df.aggregate( ... [], [dfn.functions.regr_syy( ... dfn.col("y"), dfn.col("x"), ... filter=dfn.col("y") > dfn.lit(1.0) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 0.5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.regr_syy(y.expr, x.expr, filter=filter_raw)) def first_value( expression: Expr, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment = NullTreatment.RESPECT_NULLS, ) -> Expr: """Returns the first value in a group of values. This aggregate function will return the first value in the partition. If using the builder functions described in ref:`_aggregation` this function ignores the option ``distinct``. Args: expression: Argument to perform bitwise calculation on filter: If provided, only compute against rows for which the filter is True order_by: Set the ordering of the expression to evaluate. Accepts column names or expressions. null_treatment: Assign whether to respect or ignore null values. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30]}) >>> result = df.aggregate( ... [], [dfn.functions.first_value( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 10 >>> df = ctx.from_pydict({"a": [None, 20, 10]}) >>> result = df.aggregate( ... [], [dfn.functions.first_value( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(10), ... order_by="a", ... null_treatment=dfn.common.NullTreatment.IGNORE_NULLS, ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 20 """ order_by_raw = sort_list_to_raw_sort_list(order_by) filter_raw = filter.expr if filter is not None else None return Expr( f.first_value( expression.expr, filter=filter_raw, order_by=order_by_raw, null_treatment=null_treatment.value, ) ) def last_value( expression: Expr, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment = NullTreatment.RESPECT_NULLS, ) -> Expr: """Returns the last value in a group of values. This aggregate function will return the last value in the partition. If using the builder functions described in ref:`_aggregation` this function ignores the option ``distinct``. Args: expression: Argument to perform bitwise calculation on filter: If provided, only compute against rows for which the filter is True order_by: Set the ordering of the expression to evaluate. Accepts column names or expressions. null_treatment: Assign whether to respect or ignore null values. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30]}) >>> result = df.aggregate( ... [], [dfn.functions.last_value( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 30 >>> df = ctx.from_pydict({"a": [None, 20, 10]}) >>> result = df.aggregate( ... [], [dfn.functions.last_value( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(10), ... order_by="a", ... null_treatment=dfn.common.NullTreatment.IGNORE_NULLS, ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 20 """ order_by_raw = sort_list_to_raw_sort_list(order_by) filter_raw = filter.expr if filter is not None else None return Expr( f.last_value( expression.expr, filter=filter_raw, order_by=order_by_raw, null_treatment=null_treatment.value, ) ) def nth_value( expression: Expr, n: int, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment = NullTreatment.RESPECT_NULLS, ) -> Expr: """Returns the n-th value in a group of values. This aggregate function will return the n-th value in the partition. If using the builder functions described in ref:`_aggregation` this function ignores the option ``distinct``. Args: expression: Argument to perform bitwise calculation on n: Index of value to return. Starts at 1. filter: If provided, only compute against rows for which the filter is True order_by: Set the ordering of the expression to evaluate. Accepts column names or expressions. null_treatment: Assign whether to respect or ignore null values. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30]}) >>> result = df.aggregate( ... [], [dfn.functions.nth_value( ... dfn.col("a"), 1 ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 10 >>> result = df.aggregate( ... [], [dfn.functions.nth_value( ... dfn.col("a"), 1, ... filter=dfn.col("a") > dfn.lit(10), ... order_by="a", ... null_treatment=dfn.common.NullTreatment.IGNORE_NULLS, ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 20 """ order_by_raw = sort_list_to_raw_sort_list(order_by) filter_raw = filter.expr if filter is not None else None return Expr( f.nth_value( expression.expr, n, filter=filter_raw, order_by=order_by_raw, null_treatment=null_treatment.value, ) ) def bit_and(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the bitwise AND of the argument. This aggregate function will bitwise compare every value in the input partition. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: Argument to perform bitwise calculation on filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [7, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.bit_and( ... dfn.col("a") ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 3 >>> df = ctx.from_pydict({"a": [7, 5, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.bit_and( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(3) ... ).alias("v")]) >>> result.collect_column("v")[0].as_py() 5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.bit_and(expression.expr, filter=filter_raw)) def bit_or(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the bitwise OR of the argument. This aggregate function will bitwise compare every value in the input partition. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: Argument to perform bitwise calculation on filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2]}) >>> result = df.aggregate( ... [], [dfn.functions.bit_or( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 3 >>> df = ctx.from_pydict({"a": [1, 2, 4]}) >>> result = df.aggregate( ... [], [dfn.functions.bit_or( ... dfn.col("a"), ... filter=dfn.col("a") > dfn.lit(1) ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 6 """ filter_raw = filter.expr if filter is not None else None return Expr(f.bit_or(expression.expr, filter=filter_raw)) def bit_xor( expression: Expr, distinct: bool = False, filter: Expr | None = None ) -> Expr: """Computes the bitwise XOR of the argument. This aggregate function will bitwise compare every value in the input partition. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by`` and ``null_treatment``. Args: expression: Argument to perform bitwise calculation on distinct: If True, evaluate each unique value of expression only once filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [5, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.bit_xor( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 6 >>> df = ctx.from_pydict({"a": [5, 5, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.bit_xor( ... dfn.col("a"), distinct=True, ... filter=dfn.col("a") > dfn.lit(3), ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() 5 """ filter_raw = filter.expr if filter is not None else None return Expr(f.bit_xor(expression.expr, distinct=distinct, filter=filter_raw)) def bool_and(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the boolean AND of the argument. This aggregate function will compare every value in the input partition. These are expected to be boolean values. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: Argument to perform calculation on filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [True, True, False]}) >>> result = df.aggregate( ... [], [dfn.functions.bool_and( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() False >>> df = ctx.from_pydict( ... {"a": [True, True, False], "b": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.bool_and( ... dfn.col("a"), ... filter=dfn.col("b") < dfn.lit(3) ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() True """ filter_raw = filter.expr if filter is not None else None return Expr(f.bool_and(expression.expr, filter=filter_raw)) def bool_or(expression: Expr, filter: Expr | None = None) -> Expr: """Computes the boolean OR of the argument. This aggregate function will compare every value in the input partition. These are expected to be boolean values. If using the builder functions described in ref:`_aggregation` this function ignores the options ``order_by``, ``null_treatment``, and ``distinct``. Args: expression: Argument to perform calculation on filter: If provided, only compute against rows for which the filter is True Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [False, False, True]}) >>> result = df.aggregate( ... [], [dfn.functions.bool_or( ... dfn.col("a") ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() True >>> df = ctx.from_pydict( ... {"a": [False, False, True], "b": [1, 2, 3]}) >>> result = df.aggregate( ... [], [dfn.functions.bool_or( ... dfn.col("a"), ... filter=dfn.col("b") < dfn.lit(3) ... ).alias("v")] ... ) >>> result.collect_column("v")[0].as_py() False """ filter_raw = filter.expr if filter is not None else None return Expr(f.bool_or(expression.expr, filter=filter_raw)) def lead( arg: Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a lead window function. Lead operation will return the argument that is in the next shift_offset-th row in the partition. For example ``lead(col("b"), shift_offset=3, default_value=5)`` will return the 3rd following value in column ``b``. At the end of the partition, where no further values can be returned it will return the default value of 5. Here is an example of both the ``lead`` and :py:func:`datafusion.functions.lag` functions on a simple DataFrame:: +--------+------+-----+ | points | lead | lag | +--------+------+-----+ | 100 | 100 | | | 100 | 50 | 100 | | 50 | 25 | 100 | | 25 | | 50 | +--------+------+-----+ To set window function parameters use the window builder approach described in the ref:`_window_functions` online documentation. Args: arg: Value to return shift_offset: Number of rows following the current row. default_value: Value to return if shift_offet row does not exist. partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.lead( ... dfn.col("a"), shift_offset=1, ... default_value=0, order_by="a" ... ).alias("lead")) >>> result.sort(dfn.col("a")).collect_column("lead").to_pylist() [2, 3, 0] >>> df = ctx.from_pydict({"g": ["a", "a", "b"], "v": [1, 2, 3]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.lead( ... dfn.col("v"), shift_offset=1, default_value=0, ... partition_by=dfn.col("g"), order_by="v", ... ).alias("lead")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("lead").to_pylist() [2, 0, 0] """ if not isinstance(default_value, pa.Scalar) and default_value is not None: default_value = pa.scalar(default_value) partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.lead( arg.expr, shift_offset, default_value, partition_by=partition_by_raw, order_by=order_by_raw, ) ) def lag( arg: Expr, shift_offset: int = 1, default_value: Any | None = None, partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a lag window function. Lag operation will return the argument that is in the previous shift_offset-th row in the partition. For example ``lag(col("b"), shift_offset=3, default_value=5)`` will return the 3rd previous value in column ``b``. At the beginning of the partition, where no values can be returned it will return the default value of 5. Here is an example of both the ``lag`` and :py:func:`datafusion.functions.lead` functions on a simple DataFrame:: +--------+------+-----+ | points | lead | lag | +--------+------+-----+ | 100 | 100 | | | 100 | 50 | 100 | | 50 | 25 | 100 | | 25 | | 50 | +--------+------+-----+ Args: arg: Value to return shift_offset: Number of rows before the current row. default_value: Value to return if shift_offet row does not exist. partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.lag( ... dfn.col("a"), shift_offset=1, ... default_value=0, order_by="a" ... ).alias("lag")) >>> result.sort(dfn.col("a")).collect_column("lag").to_pylist() [0, 1, 2] >>> df = ctx.from_pydict({"g": ["a", "a", "b"], "v": [1, 2, 3]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.lag( ... dfn.col("v"), shift_offset=1, default_value=0, ... partition_by=dfn.col("g"), order_by="v", ... ).alias("lag")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("lag").to_pylist() [0, 1, 0] """ if not isinstance(default_value, pa.Scalar): default_value = pa.scalar(default_value) partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.lag( arg.expr, shift_offset, default_value, partition_by=partition_by_raw, order_by=order_by_raw, ) ) def row_number( partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a row number window function. Returns the row number of the window function. Here is an example of the ``row_number`` on a simple DataFrame:: +--------+------------+ | points | row number | +--------+------------+ | 100 | 1 | | 100 | 2 | | 50 | 3 | | 25 | 4 | +--------+------------+ Args: partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.row_number( ... order_by="a" ... ).alias("rn")) >>> result.sort(dfn.col("a")).collect_column("rn").to_pylist() [1, 2, 3] >>> df = ctx.from_pydict( ... {"g": ["a", "a", "b", "b"], "v": [1, 2, 3, 4]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.row_number( ... partition_by=dfn.col("g"), order_by="v", ... ).alias("rn")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("rn").to_pylist() [1, 2, 1, 2] """ partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.row_number( partition_by=partition_by_raw, order_by=order_by_raw, ) ) def rank( partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a rank window function. Returns the rank based upon the window order. Consecutive equal values will receive the same rank, but the next different value will not be consecutive but rather the number of rows that precede it plus one. This is similar to Olympic medals. If two people tie for gold, the next place is bronze. There would be no silver medal. Here is an example of a dataframe with a window ordered by descending ``points`` and the associated rank. You should set ``order_by`` to produce meaningful results:: +--------+------+ | points | rank | +--------+------+ | 100 | 1 | | 100 | 1 | | 50 | 3 | | 25 | 4 | +--------+------+ Args: partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 10, 20]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.rank( ... order_by="a" ... ).alias("rnk") ... ) >>> result.sort(dfn.col("a")).collect_column("rnk").to_pylist() [1, 1, 3] >>> df = ctx.from_pydict( ... {"g": ["a", "a", "b", "b"], "v": [1, 1, 2, 3]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.rank( ... partition_by=dfn.col("g"), order_by="v", ... ).alias("rnk")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("rnk").to_pylist() [1, 1, 1, 2] """ partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.rank( partition_by=partition_by_raw, order_by=order_by_raw, ) ) def dense_rank( partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a dense_rank window function. This window function is similar to :py:func:`rank` except that the returned values will be consecutive. Here is an example of a dataframe with a window ordered by descending ``points`` and the associated dense rank:: +--------+------------+ | points | dense_rank | +--------+------------+ | 100 | 1 | | 100 | 1 | | 50 | 2 | | 25 | 3 | +--------+------------+ Args: partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 10, 20]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.dense_rank( ... order_by="a" ... ).alias("dr")) >>> result.sort(dfn.col("a")).collect_column("dr").to_pylist() [1, 1, 2] >>> df = ctx.from_pydict( ... {"g": ["a", "a", "b", "b"], "v": [1, 1, 2, 3]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.dense_rank( ... partition_by=dfn.col("g"), order_by="v", ... ).alias("dr")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("dr").to_pylist() [1, 1, 1, 2] """ partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.dense_rank( partition_by=partition_by_raw, order_by=order_by_raw, ) ) def percent_rank( partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a percent_rank window function. This window function is similar to :py:func:`rank` except that the returned values are the percentage from 0.0 to 1.0 from first to last. Here is an example of a dataframe with a window ordered by descending ``points`` and the associated percent rank:: +--------+--------------+ | points | percent_rank | +--------+--------------+ | 100 | 0.0 | | 100 | 0.0 | | 50 | 0.666667 | | 25 | 1.0 | +--------+--------------+ Args: partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.percent_rank( ... order_by="a" ... ).alias("pr")) >>> result.sort(dfn.col("a")).collect_column("pr").to_pylist() [0.0, 0.5, 1.0] >>> df = ctx.from_pydict( ... {"g": ["a", "a", "a", "b", "b"], "v": [1, 2, 3, 4, 5]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.percent_rank( ... partition_by=dfn.col("g"), order_by="v", ... ).alias("pr")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("pr").to_pylist() [0.0, 0.5, 1.0, 0.0, 1.0] """ partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.percent_rank( partition_by=partition_by_raw, order_by=order_by_raw, ) ) def cume_dist( partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a cumulative distribution window function. This window function is similar to :py:func:`rank` except that the returned values are the ratio of the row number to the total number of rows. Here is an example of a dataframe with a window ordered by descending ``points`` and the associated cumulative distribution:: +--------+-----------+ | points | cume_dist | +--------+-----------+ | 100 | 0.5 | | 100 | 0.5 | | 50 | 0.75 | | 25 | 1.0 | +--------+-----------+ Args: partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1., 2., 2., 3.]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.cume_dist( ... order_by="a" ... ).alias("cd") ... ) >>> result.collect_column("cd").to_pylist() [0.25..., 0.75..., 0.75..., 1.0...] >>> df = ctx.from_pydict( ... {"g": ["a", "a", "b", "b"], "v": [1, 2, 3, 4]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.cume_dist( ... partition_by=dfn.col("g"), order_by="v", ... ).alias("cd")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("cd").to_pylist() [0.5, 1.0, 0.5, 1.0] """ partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.cume_dist( partition_by=partition_by_raw, order_by=order_by_raw, ) ) def ntile( groups: int, partition_by: list[Expr] | Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Create a n-tile window function. This window function orders the window frame into a give number of groups based on the ordering criteria. It then returns which group the current row is assigned to. Here is an example of a dataframe with a window ordered by descending ``points`` and the associated n-tile function:: +--------+-------+ | points | ntile | +--------+-------+ | 120 | 1 | | 100 | 1 | | 80 | 2 | | 60 | 2 | | 40 | 3 | | 20 | 3 | +--------+-------+ Args: groups: Number of groups for the n-tile to be divided into. partition_by: Expressions to partition the window frame on. order_by: Set ordering within the window frame. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30, 40]}) >>> result = df.select( ... dfn.col("a"), ... dfn.functions.ntile( ... 2, order_by="a" ... ).alias("nt")) >>> result.sort(dfn.col("a")).collect_column("nt").to_pylist() [1, 1, 2, 2] >>> df = ctx.from_pydict( ... {"g": ["a", "a", "b", "b"], "v": [1, 2, 3, 4]}) >>> result = df.select( ... dfn.col("g"), dfn.col("v"), ... dfn.functions.ntile( ... 2, partition_by=dfn.col("g"), order_by="v", ... ).alias("nt")) >>> result.sort(dfn.col("g"), dfn.col("v")).collect_column("nt").to_pylist() [1, 2, 1, 2] """ partition_by_raw = expr_list_to_raw_expr_list(partition_by) order_by_raw = sort_list_to_raw_sort_list(order_by) return Expr( f.ntile( Expr.literal(groups).expr, partition_by=partition_by_raw, order_by=order_by_raw, ) ) def string_agg( expression: Expr, delimiter: str, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, ) -> Expr: """Concatenates the input strings. This aggregate function will concatenate input strings, ignoring null values, and separating them with the specified delimiter. Non-string values will be converted to their string equivalents. If using the builder functions described in ref:`_aggregation` this function ignores the options ``distinct`` and ``null_treatment``. Args: expression: Argument to perform bitwise calculation on delimiter: Text to place between each value of expression filter: If provided, only compute against rows for which the filter is True order_by: Set the ordering of the expression to evaluate. Accepts column names or expressions. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": ["x", "y", "z"]}) >>> result = df.aggregate( ... [], [dfn.functions.string_agg( ... dfn.col("a"), ",", order_by="a" ... ).alias("s")]) >>> result.collect_column("s")[0].as_py() 'x,y,z' >>> result = df.aggregate( ... [], [dfn.functions.string_agg( ... dfn.col("a"), ",", ... filter=dfn.col("a") > dfn.lit("x"), ... order_by="a", ... ).alias("s")]) >>> result.collect_column("s")[0].as_py() 'y,z' """ order_by_raw = sort_list_to_raw_sort_list(order_by) filter_raw = filter.expr if filter is not None else None return Expr( f.string_agg( expression.expr, delimiter, filter=filter_raw, order_by=order_by_raw, ) ) apache-datafusion-python-54.0.0/python/datafusion/functions/spark.py000066400000000000000000001627541521727353300257020ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Spark-compatible function bindings. These functions mirror the semantics of their Apache Spark counterparts exactly. Some override DataFusion built-ins (``substring`` is 1-indexed, ``concat`` propagates NULL, ``round`` uses HALF_UP rounding, etc.), which is why they live in a separate namespace rather than replacing the defaults. For DataFrame use, import this module and call functions directly. For SQL use, call :py:meth:`datafusion.SessionContext.enable_spark_functions` to register the Spark UDFs by name (overriding any built-ins with matching names) before issuing SQL queries. """ from __future__ import annotations from typing import TYPE_CHECKING, Any import pyarrow as pa from datafusion._internal import functions as _functions from datafusion.expr import ( Expr, _to_raw_expr, coerce_to_expr, sort_list_to_raw_sort_list, ) if TYPE_CHECKING: from datafusion.common import NullTreatment from datafusion.expr import SortKey _f = _functions.spark # Reused int32 literal so optional-arg defaults don't rebuild it per call. _ZERO_I32 = Expr.literal(pa.scalar(0, type=pa.int32())) def _filter_raw(filter: Expr | None) -> Any: return filter.expr if filter is not None else None def _coerce_i32(value: Expr | int | None) -> Expr | None: """Coerce a native ``int`` to an int32 literal, passing ``Expr``/``None`` through. Several Spark datetime and interval builders require 32-bit integer inputs, so a bare ``int`` must become an int32 literal rather than the int64 default that :meth:`Expr.literal` would produce. """ if value is None or isinstance(value, Expr): return value return Expr.literal(pa.scalar(value, type=pa.int32())) # --------------------------------------------------------------------------- # Aggregate functions # --------------------------------------------------------------------------- def avg( col: Expr, distinct: bool | None = None, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment | None = None, ) -> Expr: """Spark ``avg``: returns the mean of a numeric column. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) >>> r = df.aggregate( ... [], [dfn.functions.spark.avg(dfn.col("a")).alias("v")]) >>> r.collect_column("v")[0].as_py() 2.0 """ return Expr( _f.avg( col.expr, distinct=distinct, filter=_filter_raw(filter), order_by=sort_list_to_raw_sort_list(order_by), null_treatment=null_treatment, ) ) def try_sum( col: Expr, distinct: bool | None = None, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment | None = None, ) -> Expr: """Spark ``try_sum``: sum that returns NULL on overflow. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 3]}) >>> r = df.aggregate( ... [], [dfn.functions.spark.try_sum(dfn.col("a")).alias("v")]) >>> r.collect_column("v")[0].as_py() 6 """ return Expr( _f.try_sum( col.expr, distinct=distinct, filter=_filter_raw(filter), order_by=sort_list_to_raw_sort_list(order_by), null_treatment=null_treatment, ) ) def collect_list( col: Expr, distinct: bool | None = None, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment | None = None, ) -> Expr: """Spark ``collect_list``: collect values into an array (preserves dups). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 2]}) >>> r = df.aggregate( ... [], [dfn.functions.spark.collect_list(dfn.col("a")).alias("v")]) >>> sorted(r.collect_column("v")[0].as_py()) [1, 2, 2] """ return Expr( _f.collect_list( col.expr, distinct=distinct, filter=_filter_raw(filter), order_by=sort_list_to_raw_sort_list(order_by), null_treatment=null_treatment, ) ) def collect_set( col: Expr, distinct: bool | None = None, filter: Expr | None = None, order_by: list[SortKey] | SortKey | None = None, null_treatment: NullTreatment | None = None, ) -> Expr: """Spark ``collect_set``: collect distinct values into an array. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1, 2, 2, 3]}) >>> r = df.aggregate( ... [], [dfn.functions.spark.collect_set(dfn.col("a")).alias("v")]) >>> sorted(r.collect_column("v")[0].as_py()) [1, 2, 3] """ return Expr( _f.collect_set( col.expr, distinct=distinct, filter=_filter_raw(filter), order_by=sort_list_to_raw_sort_list(order_by), null_treatment=null_treatment, ) ) # --------------------------------------------------------------------------- # Array functions # --------------------------------------------------------------------------- def array_contains(col: Expr, value: Expr | Any) -> Expr: """Spark ``array_contains``: true if the array contains the element. ``value`` accepts a native Python literal or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.array_contains( ... dfn.functions.spark.array(dfn.lit(1), dfn.lit(2)), 1 ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() True """ return Expr(_f.array_contains(col.expr, coerce_to_expr(value).expr)) def array(*cols: Expr) -> Expr: """Spark ``array``: builds an array from the given elements. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.array( ... dfn.lit(1), dfn.lit(2), dfn.lit(3) ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() [1, 2, 3] """ return Expr(_f.array(*[c.expr for c in cols])) def shuffle(col: Expr, seed: int | None = None) -> Expr: """Spark ``shuffle``: returns a random permutation of the input array. ``seed`` is accepted for pyspark parity but is not yet wired through the Rust binding; passing a non-``None`` value raises ``NotImplementedError``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.shuffle( ... dfn.functions.spark.array(dfn.lit(1), dfn.lit(2), dfn.lit(3)) ... ).alias("v") ... ) >>> sorted(r.collect_column("v")[0].as_py()) [1, 2, 3] """ if seed is not None: msg = "shuffle(seed=...) is not yet supported by the Spark UDF binding" raise NotImplementedError(msg) return Expr(_f.shuffle(col.expr)) def array_repeat(col: Expr, count: Expr | int) -> Expr: """Spark ``array_repeat``: array of ``element`` repeated ``count`` times. ``count`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.array_repeat(dfn.lit("a"), 3).alias("v")) >>> r.collect_column("v")[0].as_py() ['a', 'a', 'a'] """ return Expr(_f.array_repeat(col.expr, coerce_to_expr(count).expr)) def slice(x: Expr, start: Expr | int, length: Expr | int) -> Expr: """Spark ``slice``: subset of the array from 1-indexed ``start`` with ``length``. Negative ``start`` counts from the end. ``start`` and ``length`` accept native ``int`` values or :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.slice( ... dfn.functions.spark.array( ... dfn.lit(1), dfn.lit(2), dfn.lit(3), dfn.lit(4)), ... 2, 2, ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() [2, 3] """ return Expr( _f.slice(x.expr, coerce_to_expr(start).expr, coerce_to_expr(length).expr) ) # --------------------------------------------------------------------------- # Bitmap functions # --------------------------------------------------------------------------- def bitmap_count(col: Expr) -> Expr: r"""Spark ``bitmap_count``: number of set bits in a bitmap. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.bitmap_count(dfn.lit(b"\xff")).alias("v")) >>> r.collect_column("v")[0].as_py() 8 """ return Expr(_f.bitmap_count(col.expr)) def bitmap_bit_position(col: Expr) -> Expr: """Spark ``bitmap_bit_position``: bit position for a child expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.bitmap_bit_position(dfn.lit(15)).alias("v")) >>> r.collect_column("v")[0].as_py() 14 """ return Expr(_f.bitmap_bit_position(col.expr)) def bitmap_bucket_number(col: Expr) -> Expr: """Spark ``bitmap_bucket_number``: bucket number for a child expression. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.bitmap_bucket_number(dfn.lit(15)).alias("v")) >>> r.collect_column("v")[0].as_py() 1 """ return Expr(_f.bitmap_bucket_number(col.expr)) # --------------------------------------------------------------------------- # Bitwise functions # --------------------------------------------------------------------------- def bit_get(col: Expr, pos: Expr | str) -> Expr: """Spark ``bit_get``: returns the bit (0 or 1) at ``pos``. A bare ``str`` ``pos`` is treated as a column name (matching pyspark), not a literal; pass :func:`~datafusion.lit` for a literal position. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.bit_get(dfn.lit(5), dfn.lit(0)).alias("v")) >>> r.collect_column("v")[0].as_py() 1 """ return Expr(_f.bit_get(col.expr, _to_raw_expr(pos))) def bit_count(col: Expr) -> Expr: """Spark ``bit_count``: number of bits set in the integer's binary form. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.bit_count(dfn.lit(7)).alias("v")) >>> r.collect_column("v")[0].as_py() 3 """ return Expr(_f.bit_count(col.expr)) def bitwise_not(col: Expr) -> Expr: """Spark ``~``: bitwise NOT. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.bitwise_not(dfn.lit(0)).alias("v")) >>> r.collect_column("v")[0].as_py() -1 """ return Expr(_f.bitwise_not(col.expr)) def shiftleft(col: Expr, numBits: Expr | int) -> Expr: # noqa: N803 """Spark ``shiftleft``: ``value`` shifted left by ``shift`` bits. ``numBits`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.shiftleft(dfn.lit(1), 3).alias("v")) >>> r.collect_column("v")[0].as_py() 8 """ return Expr(_f.shiftleft(col.expr, coerce_to_expr(numBits).expr)) def shiftright(col: Expr, numBits: Expr | int) -> Expr: # noqa: N803 """Spark ``shiftright``: arithmetic right shift. ``numBits`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.shiftright(dfn.lit(8), 2).alias("v")) >>> r.collect_column("v")[0].as_py() 2 """ return Expr(_f.shiftright(col.expr, coerce_to_expr(numBits).expr)) def shiftrightunsigned(col: Expr, numBits: Expr | int) -> Expr: # noqa: N803 """Spark ``shiftrightunsigned``: logical (unsigned) right shift. ``numBits`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.shiftrightunsigned(dfn.lit(8), 2).alias("v")) >>> r.collect_column("v")[0].as_py() 2 """ return Expr(_f.shiftrightunsigned(col.expr, coerce_to_expr(numBits).expr)) # --------------------------------------------------------------------------- # Collection / Conditional / Conversion # --------------------------------------------------------------------------- def size(col: Expr) -> Expr: """Spark ``size``: length of an array or map. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.size( ... dfn.functions.spark.array(dfn.lit(1), dfn.lit(2), dfn.lit(3)) ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 3 """ return Expr(_f.size(col.expr)) def if_(condition: Expr, if_true: Expr | Any, if_false: Expr | Any) -> Expr: """Spark ``if``: returns ``if_true`` when ``condition`` is true, else ``if_false``. Exposed as ``if_`` because ``if`` is a Python keyword. ``if_true`` and ``if_false`` accept native Python literals or :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.if_( ... dfn.lit(2) > dfn.lit(1), "big", "small" ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 'big' """ return Expr( _f.if_( condition.expr, coerce_to_expr(if_true).expr, coerce_to_expr(if_false).expr ) ) def spark_cast(arg: Expr, type_str: Expr | str) -> Expr: """Spark ``cast``: cast ``arg`` to the type named by ``type_str``. Uses Spark cast semantics (e.g. overflow returns NULL, not error). ``type_str`` accepts a native ``str`` or an :class:`Expr`. Currently only supports casting numeric values to ``"timestamp"``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.spark_cast( ... dfn.lit(1579098645), "timestamp" ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py().isoformat() '2020-01-15T14:30:45+00:00' """ return Expr(_f.spark_cast(arg.expr, coerce_to_expr(type_str).expr)) # --------------------------------------------------------------------------- # Datetime functions # --------------------------------------------------------------------------- def add_months(start: Expr, months: Expr | int) -> Expr: """Spark ``add_months``: date + N months. ``months`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.add_months(d, 2).alias("v")) >>> r.collect_column("v")[0].as_py() datetime.date(2020, 3, 15) """ return Expr(_f.add_months(start.expr, _coerce_i32(months).expr)) def date_add(start: Expr, days: Expr | int) -> Expr: """Spark ``date_add``: date + N days. ``days`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.date_add(d, 5).alias("v")) >>> r.collect_column("v")[0].as_py() datetime.date(2020, 1, 20) """ return Expr(_f.date_add(start.expr, _coerce_i32(days).expr)) def date_sub(start: Expr, days: Expr | int) -> Expr: """Spark ``date_sub``: date - N days. ``days`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.date_sub(d, 5).alias("v")) >>> r.collect_column("v")[0].as_py() datetime.date(2020, 1, 10) """ return Expr(_f.date_sub(start.expr, _coerce_i32(days).expr)) def hour(col: Expr) -> Expr: """Spark ``hour``: extract hour component of a timestamp. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select(dfn.functions.spark.hour(ts).alias("v")) >>> r.collect_column("v")[0].as_py() 14 """ return Expr(_f.hour(col.expr)) def minute(col: Expr) -> Expr: """Spark ``minute``: extract minute component of a timestamp. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select(dfn.functions.spark.minute(ts).alias("v")) >>> r.collect_column("v")[0].as_py() 30 """ return Expr(_f.minute(col.expr)) def second(col: Expr) -> Expr: """Spark ``second``: extract second component of a timestamp. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select(dfn.functions.spark.second(ts).alias("v")) >>> r.collect_column("v")[0].as_py() 45 """ return Expr(_f.second(col.expr)) def last_day(col: Expr) -> Expr: """Spark ``last_day``: last day of the month containing the date. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.last_day(d).alias("v")) >>> r.collect_column("v")[0].as_py() datetime.date(2020, 1, 31) """ return Expr(_f.last_day(col.expr)) def make_dt_interval( days: Expr | int | None = None, hours: Expr | int | None = None, mins: Expr | int | None = None, secs: Expr | float | None = None, ) -> Expr: """Spark ``make_dt_interval``: day-time interval from components. All parts are optional; omitted parts default to zero, matching pyspark. Integer parts accept a native ``int`` and ``secs`` accepts a ``float``, or any part may be an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.make_dt_interval().alias("v")) >>> r.collect_column("v")[0].as_py() datetime.timedelta(0) >>> r = df.select( ... dfn.functions.spark.make_dt_interval( ... days=1, hours=2, mins=3, secs=4.5 ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() datetime.timedelta(days=1, seconds=7384, microseconds=500000) """ return Expr( _f.make_dt_interval( (_coerce_i32(days) if days is not None else _ZERO_I32).expr, (_coerce_i32(hours) if hours is not None else _ZERO_I32).expr, (_coerce_i32(mins) if mins is not None else _ZERO_I32).expr, (coerce_to_expr(secs) if secs is not None else Expr.literal(0.0)).expr, ) ) def make_interval( years: Expr | int | None = None, months: Expr | int | None = None, weeks: Expr | int | None = None, days: Expr | int | None = None, hours: Expr | int | None = None, mins: Expr | int | None = None, secs: Expr | float | None = None, ) -> Expr: """Spark ``make_interval``: interval from year/month/week/day/hour/min/sec parts. All parts are optional; omitted parts default to zero, matching pyspark. Integer parts accept a native ``int`` and ``secs`` accepts a ``float``, or any part may be an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.make_interval().alias("v")) >>> r.collect_column("v")[0].as_py().months 0 >>> r = df.select(dfn.functions.spark.make_interval(years=1).alias("v")) >>> r.collect_column("v")[0].as_py().months 12 """ return Expr( _f.make_interval( (_coerce_i32(years) if years is not None else _ZERO_I32).expr, (_coerce_i32(months) if months is not None else _ZERO_I32).expr, (_coerce_i32(weeks) if weeks is not None else _ZERO_I32).expr, (_coerce_i32(days) if days is not None else _ZERO_I32).expr, (_coerce_i32(hours) if hours is not None else _ZERO_I32).expr, (_coerce_i32(mins) if mins is not None else _ZERO_I32).expr, (coerce_to_expr(secs) if secs is not None else Expr.literal(0.0)).expr, ) ) def next_day(date: Expr, dayOfWeek: Expr | str) -> Expr: # noqa: N803 """Spark ``next_day``: first date after ``start_date`` named ``day_of_week``. ``dayOfWeek`` accepts a native ``str`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.next_day(d, "Mon").alias("v")) >>> r.collect_column("v")[0].as_py() datetime.date(2020, 1, 20) """ return Expr(_f.next_day(date.expr, coerce_to_expr(dayOfWeek).expr)) def date_diff(end: Expr, start: Expr) -> Expr: """Spark ``date_diff``: number of days from ``start_date`` to ``end_date``. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> end = dfn.lit(pa.scalar(date(2020, 1, 20), type=pa.date32())) >>> r = df.select(dfn.functions.spark.date_diff(end, d).alias("v")) >>> r.collect_column("v")[0].as_py() 5 """ return Expr(_f.date_diff(end.expr, start.expr)) def date_trunc(format: Expr | str, timestamp: Expr) -> Expr: """Spark ``date_trunc``: truncate timestamp to unit ``fmt``. ``format`` accepts a native ``str`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select( ... dfn.functions.spark.date_trunc("month", ts).alias("v")) >>> r.collect_column("v")[0].as_py() datetime.datetime(2020, 1, 1, 0, 0) """ return Expr(_f.date_trunc(coerce_to_expr(format).expr, timestamp.expr)) def time_trunc(unit: Expr | str, time: Expr) -> Expr: """Spark ``time_trunc``: truncate time value to unit ``fmt``. A bare ``str`` ``unit`` is treated as a column name (matching pyspark), not a literal; pass :func:`~datafusion.lit` for a literal unit. Examples: >>> import pyarrow as pa >>> from datetime import time >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> t = dfn.lit(pa.scalar(time(14, 30, 45), type=pa.time64('us'))) >>> r = df.select( ... dfn.functions.spark.time_trunc(dfn.lit("hour"), t).alias("v")) >>> r.collect_column("v")[0].as_py() datetime.time(14, 0) """ return Expr(_f.time_trunc(_to_raw_expr(unit), time.expr)) def trunc(date: Expr, format: Expr | str) -> Expr: """Spark ``trunc``: truncate date to unit ``fmt``. ``format`` accepts a native ``str`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.trunc(d, "YEAR").alias("v")) >>> r.collect_column("v")[0].as_py() datetime.date(2020, 1, 1) """ return Expr(_f.trunc(date.expr, coerce_to_expr(format).expr)) def date_part(field: Expr | str, source: Expr) -> Expr: """Spark ``date_part``: extract ``field`` from a date/time/timestamp. ``field`` accepts a native ``str`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select( ... dfn.functions.spark.date_part("year", d).alias("v")) >>> r.collect_column("v")[0].as_py() 2020 """ return Expr(_f.date_part(coerce_to_expr(field).expr, source.expr)) def from_utc_timestamp(timestamp: Expr, tz: Expr | str) -> Expr: """Spark ``from_utc_timestamp``: interpret ``ts`` as UTC, convert to ``tz``. ``tz`` accepts a native ``str`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select( ... dfn.functions.spark.from_utc_timestamp(ts, "UTC").alias("v")) >>> r.collect_column("v")[0].as_py() datetime.datetime(2020, 1, 15, 14, 30, 45) """ return Expr(_f.from_utc_timestamp(timestamp.expr, coerce_to_expr(tz).expr)) def to_utc_timestamp(timestamp: Expr, tz: Expr | str) -> Expr: """Spark ``to_utc_timestamp``: interpret ``ts`` as ``tz``, convert to UTC. ``tz`` accepts a native ``str`` or an :class:`Expr`. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select( ... dfn.functions.spark.to_utc_timestamp(ts, "UTC").alias("v")) >>> r.collect_column("v")[0].as_py() datetime.datetime(2020, 1, 15, 14, 30, 45) """ return Expr(_f.to_utc_timestamp(timestamp.expr, coerce_to_expr(tz).expr)) def unix_date(col: Expr) -> Expr: """Spark ``unix_date``: days since 1970-01-01 for ``dt``. Examples: >>> import pyarrow as pa >>> from datetime import date >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> d = dfn.lit(pa.scalar(date(2020, 1, 15), type=pa.date32())) >>> r = df.select(dfn.functions.spark.unix_date(d).alias("v")) >>> r.collect_column("v")[0].as_py() 18276 """ return Expr(_f.unix_date(col.expr)) def unix_micros(col: Expr) -> Expr: """Spark ``unix_micros``: microseconds since epoch for ``ts``. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select(dfn.functions.spark.unix_micros(ts).alias("v")) >>> r.collect_column("v")[0].as_py() 1579098645000000 """ return Expr(_f.unix_micros(col.expr)) def unix_millis(col: Expr) -> Expr: """Spark ``unix_millis``: milliseconds since epoch for ``ts``. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select(dfn.functions.spark.unix_millis(ts).alias("v")) >>> r.collect_column("v")[0].as_py() 1579098645000 """ return Expr(_f.unix_millis(col.expr)) def unix_seconds(col: Expr) -> Expr: """Spark ``unix_seconds``: seconds since epoch for ``ts``. Examples: >>> import pyarrow as pa >>> from datetime import datetime >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> ts = dfn.lit( ... pa.scalar(datetime(2020, 1, 15, 14, 30, 45), ... type=pa.timestamp('us'))) >>> r = df.select(dfn.functions.spark.unix_seconds(ts).alias("v")) >>> r.collect_column("v")[0].as_py() 1579098645 """ return Expr(_f.unix_seconds(col.expr)) # --------------------------------------------------------------------------- # Hash functions # --------------------------------------------------------------------------- def crc32(col: Expr) -> Expr: """Spark ``crc32``: cyclic redundancy check value as a bigint. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"s": ["ABC"]}) >>> r = df.select(dfn.functions.spark.crc32(dfn.col("s")).alias("v")) >>> r.collect_column("v")[0].as_py() 2743272264 """ return Expr(_f.crc32(col.expr)) def sha1(col: Expr) -> Expr: """Spark ``sha1``: SHA-1 hash as a hex string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"s": ["hello"]}) >>> r = df.select(dfn.functions.spark.sha1(dfn.col("s")).alias("v")) >>> r.collect_column("v")[0].as_py() 'aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d' """ return Expr(_f.sha1(col.expr)) def sha2(col: Expr, numBits: Expr | int) -> Expr: # noqa: N803 """Spark ``sha2``: SHA-2 family hash (224, 256, 384, 512). Bit length 0 = 256. ``numBits`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"s": ["hello"]}) >>> r = df.select( ... dfn.functions.spark.sha2(dfn.col("s"), 256).alias("v")) >>> r.collect_column("v")[0].as_py() '2cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824' """ return Expr(_f.sha2(col.expr, coerce_to_expr(numBits).expr)) def xxhash64(*cols: Expr) -> Expr: """Spark ``xxhash64``: 64-bit xxHash of the arguments. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.xxhash64(dfn.lit("hello")).alias("v")) >>> r.collect_column("v")[0].as_py() -4367754540140381902 """ return Expr(_f.xxhash64(*[c.expr for c in cols])) # --------------------------------------------------------------------------- # JSON functions # --------------------------------------------------------------------------- def json_tuple(col: Expr, *fields: Expr | str) -> Expr: """Spark ``json_tuple``: extract top-level fields from a JSON string. Each field name accepts a native ``str`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.json_tuple( ... dfn.lit('{"a":1,"b":"x"}'), "a", "b" ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() {'c0': '1', 'c1': 'x'} """ return Expr(_f.json_tuple(col.expr, *[coerce_to_expr(f).expr for f in fields])) # --------------------------------------------------------------------------- # Map functions # --------------------------------------------------------------------------- def map_from_arrays(col1: Expr, col2: Expr) -> Expr: """Spark ``map_from_arrays``: build a map from parallel key/value arrays. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> keys = dfn.functions.spark.array(dfn.lit("a"), dfn.lit("b")) >>> vals = dfn.functions.spark.array(dfn.lit(1), dfn.lit(2)) >>> r = df.select( ... dfn.functions.spark.map_from_arrays(keys, vals).alias("v")) >>> r.collect_column("v")[0].as_py() [('a', 1), ('b', 2)] """ return Expr(_f.map_from_arrays(col1.expr, col2.expr)) def map_from_entries(col: Expr) -> Expr: """Spark ``map_from_entries``: build a map from an array of key/value structs. ``col`` must be an array whose elements are two-field structs; the first field becomes the map key and the second the value. Examples: >>> import pyarrow as pa >>> ctx = dfn.SessionContext() >>> entry_type = pa.list_( ... pa.struct([("key", pa.string()), ("value", pa.int64())])) >>> entries = pa.array( ... [[{"key": "a", "value": 1}, {"key": "b", "value": 2}]], ... type=entry_type) >>> df = ctx.from_arrow(pa.record_batch([entries], names=["e"])) >>> r = df.select( ... dfn.functions.spark.map_from_entries(dfn.col("e")).alias("v")) >>> r.collect_column("v")[0].as_py() [('a', 1), ('b', 2)] """ return Expr(_f.map_from_entries(col.expr)) def str_to_map( text: Expr, pairDelim: Expr | str | None = None, # noqa: N803 keyValueDelim: Expr | str | None = None, # noqa: N803 ) -> Expr: """Spark ``str_to_map``: split text into key/value pairs using delimiters. Delimiters default to ``","`` and ``":"`` when omitted, matching pyspark. Parameter names match ``pyspark.sql.functions.str_to_map``; pyspark types the delimiters as column-or-name, so a bare ``str`` is treated as a column name. Pass :func:`~datafusion.lit` for a literal delimiter. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.str_to_map(dfn.lit("a:1,b:2")).alias("v")) >>> r.collect_column("v")[0].as_py() [('a', '1'), ('b', '2')] >>> r = df.select( ... dfn.functions.spark.str_to_map( ... dfn.lit("a=1;b=2"), ... pairDelim=dfn.lit(";"), ... keyValueDelim=dfn.lit("="), ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() [('a', '1'), ('b', '2')] """ pd = _to_raw_expr(pairDelim) if pairDelim is not None else Expr.literal(",").expr kvd = ( _to_raw_expr(keyValueDelim) if keyValueDelim is not None else Expr.literal(":").expr ) return Expr(_f.str_to_map(text.expr, pd, kvd)) # --------------------------------------------------------------------------- # Math functions # --------------------------------------------------------------------------- def abs(col: Expr) -> Expr: """Spark ``abs``: absolute value. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.abs(dfn.lit(-5)).alias("v")) >>> r.collect_column("v")[0].as_py() 5 """ return Expr(_f.abs(col.expr)) def ceil(col: Expr) -> Expr: """Spark ``ceil``: smallest integer ≥ arg. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.ceil(dfn.lit(1.2)).alias("v")) >>> r.collect_column("v")[0].as_py() 2 """ return Expr(_f.ceil(col.expr)) def expm1(col: Expr) -> Expr: """Spark ``expm1``: exp(arg) - 1. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.expm1(dfn.lit(0.0)).alias("v")) >>> r.collect_column("v")[0].as_py() 0.0 """ return Expr(_f.expm1(col.expr)) def factorial(col: Expr) -> Expr: """Spark ``factorial``: n! for n in [0..20], else NULL. Examples: >>> import pyarrow as pa >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.factorial( ... dfn.lit(pa.scalar(5, type=pa.int32())) ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 120 """ return Expr(_f.factorial(col.expr)) def floor(col: Expr) -> Expr: """Spark ``floor``: largest integer ≤ arg. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.floor(dfn.lit(1.8)).alias("v")) >>> r.collect_column("v")[0].as_py() 1 """ return Expr(_f.floor(col.expr)) def hex(col: Expr) -> Expr: """Spark ``hex``: hexadecimal representation. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.hex(dfn.lit(255)).alias("v")) >>> r.collect_column("v")[0].as_py() 'FF' """ return Expr(_f.hex(col.expr)) def modulus(dividend: Expr | float, divisor: Expr | float) -> Expr: """Spark ``mod``: remainder of ``dividend / divisor`` (sign follows dividend). ``dividend`` and ``divisor`` accept native numbers or :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.modulus(dfn.lit(10), dfn.lit(3)).alias("v")) >>> r.collect_column("v")[0].as_py() 1 """ return Expr(_f.modulus(coerce_to_expr(dividend).expr, coerce_to_expr(divisor).expr)) def pmod(dividend: Expr | float, divisor: Expr | float) -> Expr: """Spark ``pmod``: positive remainder of division. ``dividend`` and ``divisor`` accept native numbers or :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.pmod(dfn.lit(-1), dfn.lit(3)).alias("v")) >>> r.collect_column("v")[0].as_py() 2 """ return Expr(_f.pmod(coerce_to_expr(dividend).expr, coerce_to_expr(divisor).expr)) def rint(col: Expr) -> Expr: """Spark ``rint``: round to nearest mathematical integer (as double). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.rint(dfn.lit(2.5)).alias("v")) >>> r.collect_column("v")[0].as_py() 2.0 """ return Expr(_f.rint(col.expr)) def round(col: Expr, scale: Expr | int | None = None) -> Expr: """Spark ``round``: round to ``scale`` decimal places, HALF_UP rounding. ``scale`` defaults to zero when omitted, matching pyspark, and accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.round(dfn.lit(2.5)).alias("v")) >>> r.collect_column("v")[0].as_py() 3.0 >>> r = df.select( ... dfn.functions.spark.round(dfn.lit(2.345), scale=2).alias("v")) >>> r.collect_column("v")[0].as_py() 2.35 """ scale_expr = coerce_to_expr(scale) if scale is not None else _ZERO_I32 return Expr(_f.round(col.expr, scale_expr.expr)) def unhex(col: Expr) -> Expr: r"""Spark ``unhex``: convert hexadecimal string to binary. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.unhex(dfn.lit("FF")).alias("v")) >>> r.collect_column("v")[0].as_py() b'\xff' """ return Expr(_f.unhex(col.expr)) def width_bucket( v: Expr, min: Expr, max: Expr, numBucket: Expr | int, # noqa: N803 ) -> Expr: """Spark ``width_bucket``: bucket number for ``value`` in equi-width histogram. ``numBucket`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.width_bucket( ... dfn.lit(5.0), dfn.lit(0.0), dfn.lit(10.0), 5 ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 3 """ return Expr( _f.width_bucket(v.expr, min.expr, max.expr, coerce_to_expr(numBucket).expr) ) def csc(col: Expr) -> Expr: """Spark ``csc``: cosecant. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.csc(dfn.lit(1.5708)).alias("v")) >>> f"{r.collect_column('v')[0].as_py():.4f}" '1.0000' """ return Expr(_f.csc(col.expr)) def sec(col: Expr) -> Expr: """Spark ``sec``: secant. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.sec(dfn.lit(0.0)).alias("v")) >>> r.collect_column("v")[0].as_py() 1.0 """ return Expr(_f.sec(col.expr)) def negative(col: Expr) -> Expr: """Spark ``negative``: unary minus. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.negative(dfn.lit(3)).alias("v")) >>> r.collect_column("v")[0].as_py() -3 """ return Expr(_f.negative(col.expr)) def bin(col: Expr) -> Expr: """Spark ``bin``: binary string representation of a long. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.bin(dfn.lit(7)).alias("v")) >>> r.collect_column("v")[0].as_py() '111' """ return Expr(_f.bin(col.expr)) # --------------------------------------------------------------------------- # String functions # --------------------------------------------------------------------------- def ascii(col: Expr) -> Expr: """Spark ``ascii``: code point of the first character. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.ascii(dfn.lit("A")).alias("v")) >>> r.collect_column("v")[0].as_py() 65 """ return Expr(_f.ascii(col.expr)) def base64(col: Expr) -> Expr: """Spark ``base64``: encode binary as a base64 string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.base64(dfn.lit(b"hi")).alias("v")) >>> r.collect_column("v")[0].as_py() 'aGk=' """ return Expr(_f.base64(col.expr)) def char(col: Expr) -> Expr: """Spark ``char``: ASCII character for a code point (mod 256). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.char(dfn.lit(65)).alias("v")) >>> r.collect_column("v")[0].as_py() 'A' """ return Expr(_f.char(col.expr)) def concat(*cols: Expr) -> Expr: """Spark ``concat``: concatenates strings; NULL if any input is NULL. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.concat(dfn.lit("a"), dfn.lit("b")).alias("v")) >>> r.collect_column("v")[0].as_py() 'ab' """ return Expr(_f.concat(*[c.expr for c in cols])) def elt(*inputs: Expr) -> Expr: """Spark ``elt``: returns the n-th input (1-indexed). Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.elt( ... dfn.lit(2), dfn.lit("a"), dfn.lit("b") ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 'b' """ return Expr(_f.elt(*[i.expr for i in inputs])) def ilike( str: Expr, pattern: Expr | str, escapeChar: str | None = None, # noqa: N803 ) -> Expr: """Spark ``ilike``: case-insensitive pattern match. A bare ``str`` ``pattern`` is treated as a column name (matching pyspark), not a literal; pass :func:`~datafusion.lit` for a literal pattern. ``escapeChar`` is accepted for pyspark parity but is not yet wired through the Rust binding; passing a non-``None`` value raises ``NotImplementedError``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.ilike(dfn.lit("HELLO"), dfn.lit("h%")).alias("v")) >>> r.collect_column("v")[0].as_py() True """ if escapeChar is not None: msg = "ilike(escapeChar=...) is not yet supported by the Spark UDF binding" raise NotImplementedError(msg) return Expr(_f.ilike(str.expr, _to_raw_expr(pattern))) def length(col: Expr) -> Expr: """Spark ``length``: character length of a string, or byte length of binary. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.length(dfn.lit("hello")).alias("v")) >>> r.collect_column("v")[0].as_py() 5 """ return Expr(_f.length(col.expr)) def like( str: Expr, pattern: Expr | str, escapeChar: str | None = None, # noqa: N803 ) -> Expr: """Spark ``like``: case-sensitive pattern match. A bare ``str`` ``pattern`` is treated as a column name (matching pyspark), not a literal; pass :func:`~datafusion.lit` for a literal pattern. ``escapeChar`` is accepted for pyspark parity but is not yet wired through the Rust binding; passing a non-``None`` value raises ``NotImplementedError``. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.like(dfn.lit("hello"), dfn.lit("h%")).alias("v")) >>> r.collect_column("v")[0].as_py() True """ if escapeChar is not None: msg = "like(escapeChar=...) is not yet supported by the Spark UDF binding" raise NotImplementedError(msg) return Expr(_f.like(str.expr, _to_raw_expr(pattern))) def luhn_check(col: Expr) -> Expr: """Spark ``luhn_check``: true if the digit string passes the Luhn check. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.luhn_check( ... dfn.lit("4111111111111111") ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() True """ return Expr(_f.luhn_check(col.expr)) def format_string(format: str | Expr, *cols: Expr) -> Expr: """Spark ``format_string``: printf-style format string. ``format`` is the printf-style template (a plain ``str`` is auto-promoted to a literal expression); remaining args are values to substitute. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.format_string( ... "%d-%s", dfn.lit(42), dfn.lit("hi") ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() '42-hi' """ fmt_expr = format if isinstance(format, Expr) else Expr.literal(format) return Expr(_f.format_string(fmt_expr.expr, *[c.expr for c in cols])) def space(col: Expr | int) -> Expr: """Spark ``space``: string of n spaces. ``col`` accepts a native ``int`` or an :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.space(3).alias("v")) >>> r.collect_column("v")[0].as_py() ' ' """ return Expr(_f.space(_coerce_i32(col).expr)) def substring(str: Expr, pos: Expr | int, len: Expr | int) -> Expr: """Spark ``substring``: 1-indexed substring starting at ``pos`` of given ``length``. Negative ``pos`` counts from the end. ``pos`` and ``len`` accept native ``int`` values or :class:`Expr`. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.substring(dfn.lit("hello"), 1, 3).alias("v")) >>> r.collect_column("v")[0].as_py() 'hel' """ return Expr( _f.substring(str.expr, coerce_to_expr(pos).expr, coerce_to_expr(len).expr) ) def unbase64(col: Expr) -> Expr: """Spark ``unbase64``: decode a base64 string to binary. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.unbase64(dfn.lit("aGk=")).alias("v")) >>> r.collect_column("v")[0].as_py() b'hi' """ return Expr(_f.unbase64(col.expr)) def soundex(col: Expr) -> Expr: """Spark ``soundex``: Soundex phonetic code. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select(dfn.functions.spark.soundex(dfn.lit("Robert")).alias("v")) >>> r.collect_column("v")[0].as_py() 'R163' """ return Expr(_f.soundex(col.expr)) def is_valid_utf8(str: Expr) -> Expr: """Spark ``is_valid_utf8``: true if the string is valid UTF-8. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.is_valid_utf8(dfn.lit("hello")).alias("v")) >>> r.collect_column("v")[0].as_py() True """ return Expr(_f.is_valid_utf8(str.expr)) def make_valid_utf8(str: Expr) -> Expr: """Spark ``make_valid_utf8``: replace invalid UTF-8 bytes with U+FFFD. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.make_valid_utf8(dfn.lit("hello")).alias("v")) >>> r.collect_column("v")[0].as_py() 'hello' """ return Expr(_f.make_valid_utf8(str.expr)) # --------------------------------------------------------------------------- # URL functions # --------------------------------------------------------------------------- def parse_url( url: Expr, partToExtract: Expr | str, # noqa: N803 key: Expr | str | None = None, ) -> Expr: """Spark ``parse_url``: extract a part from a URL; errors on invalid URLs. ``partToExtract`` is one of ``"HOST"``, ``"PATH"``, ``"QUERY"``, ``"REF"``, ``"PROTOCOL"``, ``"FILE"``, ``"AUTHORITY"``, ``"USERINFO"``. Pass ``key`` only with ``"QUERY"`` to extract a single parameter. Bare ``str`` values for ``partToExtract``/``key`` are treated as column names (matching pyspark); pass :func:`~datafusion.lit` for a literal. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.parse_url( ... dfn.lit("http://example.com/path?q=1"), dfn.lit("HOST") ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 'example.com' >>> r = df.select( ... dfn.functions.spark.parse_url( ... dfn.lit("http://example.com/path?q=1"), ... dfn.lit("QUERY"), ... key=dfn.lit("q"), ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() '1' """ if key is None: return Expr(_f.parse_url(url.expr, _to_raw_expr(partToExtract))) return Expr(_f.parse_url(url.expr, _to_raw_expr(partToExtract), _to_raw_expr(key))) def try_parse_url( url: Expr, partToExtract: Expr | str, # noqa: N803 key: Expr | str | None = None, ) -> Expr: """Spark ``try_parse_url``: like ``parse_url`` but returns NULL on invalid URLs. Bare ``str`` values for ``partToExtract``/``key`` are treated as column names (matching pyspark); pass :func:`~datafusion.lit` for a literal. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.try_parse_url( ... dfn.lit("http://example.com/"), dfn.lit("HOST") ... ).alias("v") ... ) >>> r.collect_column("v")[0].as_py() 'example.com' """ if key is None: return Expr(_f.try_parse_url(url.expr, _to_raw_expr(partToExtract))) return Expr( _f.try_parse_url(url.expr, _to_raw_expr(partToExtract), _to_raw_expr(key)) ) def url_decode(str: Expr) -> Expr: """Spark ``url_decode``: decode an application/x-www-form-urlencoded string. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.url_decode(dfn.lit("a%20b")).alias("v")) >>> r.collect_column("v")[0].as_py() 'a b' """ return Expr(_f.url_decode(str.expr)) def try_url_decode(str: Expr) -> Expr: """Spark ``try_url_decode``: like ``url_decode``; returns NULL on invalid input. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.try_url_decode(dfn.lit("a%20b")).alias("v")) >>> r.collect_column("v")[0].as_py() 'a b' """ return Expr(_f.try_url_decode(str.expr)) def url_encode(str: Expr) -> Expr: """Spark ``url_encode``: encode a string in application/x-www-form-urlencoded. Examples: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1]}) >>> r = df.select( ... dfn.functions.spark.url_encode(dfn.lit("a b")).alias("v")) >>> r.collect_column("v")[0].as_py() 'a+b' """ return Expr(_f.url_encode(str.expr)) __all__ = [ # Math "abs", # Datetime "add_months", "array", # Array "array_contains", "array_repeat", # String "ascii", # Aggregate "avg", "base64", "bin", "bit_count", # Bitwise "bit_get", "bitmap_bit_position", "bitmap_bucket_number", # Bitmap "bitmap_count", "bitwise_not", "ceil", "char", "collect_list", "collect_set", "concat", # Hash "crc32", "csc", "date_add", "date_diff", "date_part", "date_sub", "date_trunc", "elt", "expm1", "factorial", "floor", "format_string", "from_utc_timestamp", "hex", "hour", "if_", "ilike", "is_valid_utf8", # JSON "json_tuple", "last_day", "length", "like", "luhn_check", "make_dt_interval", "make_interval", "make_valid_utf8", # Map "map_from_arrays", "map_from_entries", "minute", "modulus", "negative", "next_day", # URL "parse_url", "pmod", "rint", "round", "sec", "second", "sha1", "sha2", "shiftleft", "shiftright", "shiftrightunsigned", "shuffle", # Collection / Conditional / Conversion "size", "slice", "soundex", "space", "spark_cast", "str_to_map", "substring", "time_trunc", "to_utc_timestamp", "trunc", "try_parse_url", "try_sum", "try_url_decode", "unbase64", "unhex", "unix_date", "unix_micros", "unix_millis", "unix_seconds", "url_decode", "url_encode", "width_bucket", "xxhash64", ] apache-datafusion-python-54.0.0/python/datafusion/input/000077500000000000000000000000001521727353300233205ustar00rootroot00000000000000apache-datafusion-python-54.0.0/python/datafusion/input/__init__.py000066400000000000000000000017351521727353300254370ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """This package provides for input sources. The primary class used within DataFusion is ``LocationInputPlugin``. """ from .location import LocationInputPlugin __all__ = [ "LocationInputPlugin", ] apache-datafusion-python-54.0.0/python/datafusion/input/base.py000066400000000000000000000033261521727353300246100ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """This module provides ``BaseInputSource``. A user can extend this to provide a custom input source. """ from abc import ABC, abstractmethod from typing import Any from datafusion.common import SqlTable class BaseInputSource(ABC): """Base Input Source class. If a consuming library would like to provider their own InputSource this is the class they should extend to write their own. Once completed the Plugin InputSource can be registered with the SessionContext to ensure that it will be used in order to obtain the SqlTable information from the custom datasource. """ @abstractmethod def is_correct_input(self, input_item: Any, table_name: str, **kwargs: Any) -> bool: """Returns `True` if the input is valid.""" @abstractmethod def build_table(self, input_item: Any, table_name: str, **kwarg: Any) -> SqlTable: # type: ignore[invalid-type-form] """Create a table from the input source.""" apache-datafusion-python-54.0.0/python/datafusion/input/location.py000066400000000000000000000067621521727353300255150ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """The default input source for DataFusion.""" from pathlib import Path from typing import Any from datafusion.common import DataTypeMap, SqlTable from datafusion.input.base import BaseInputSource class LocationInputPlugin(BaseInputSource): """Input Plugin for everything. This can be read in from a file (on disk, remote etc.). """ def is_correct_input(self, input_item: Any, table_name: str, **kwargs: Any) -> bool: # noqa: ARG002 """Returns `True` if the input is valid.""" return isinstance(input_item, str) def build_table( self, input_item: str, table_name: str, **kwargs: Any, # noqa: ARG002 ) -> SqlTable: # type: ignore[invalid-type-form] """Create a table from the input source.""" extension = Path(input_item).suffix file_format = extension.lstrip(".").lower() num_rows = 0 # Total number of rows in the file. Used for statistics columns = [] if file_format == "parquet": import pyarrow.parquet as pq # noqa: PLC0415 # Read the Parquet metadata metadata = pq.read_metadata(input_item) num_rows = metadata.num_rows # Iterate through the schema and build the SqlTable columns = [ ( col.name, DataTypeMap.from_parquet_type_str(col.physical_type), ) for col in metadata.schema ] elif format == "csv": import csv # noqa: PLC0415 # Consume header row and count number of rows for statistics. # TODO: Possibly makes sense to have the eager number of rows # calculated as a configuration since you must read the entire file # to get that information. However, this should only be occurring # at table creation time and therefore shouldn't # slow down query performance. with Path(input_item).open() as file: reader = csv.reader(file) _header_row = next(reader) for _ in reader: num_rows += 1 # TODO: Need to actually consume this row into reasonable columns msg = "TODO: Currently unable to support CSV input files." raise RuntimeError(msg) else: msg = f"Input of format: `{format}` is currently not supported.\ Only Parquet and CSV." raise RuntimeError(msg) # Input could possibly be multiple files. Create a list if so input_path = Path(input_item) input_files = [str(p) for p in input_path.parent.glob(input_path.name)] return SqlTable(table_name, columns, num_rows, input_files) apache-datafusion-python-54.0.0/python/datafusion/io.py000066400000000000000000000154611521727353300231510ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """IO read functions using global context.""" from __future__ import annotations from typing import TYPE_CHECKING from datafusion.context import SessionContext if TYPE_CHECKING: import pathlib import pyarrow as pa from datafusion.dataframe import DataFrame from datafusion.expr import Expr from .options import CsvReadOptions def read_parquet( path: str | pathlib.Path, table_partition_cols: list[tuple[str, str | pa.DataType]] | None = None, parquet_pruning: bool = True, file_extension: str = ".parquet", skip_metadata: bool = True, schema: pa.Schema | None = None, file_sort_order: list[list[Expr]] | None = None, ) -> DataFrame: """Read a Parquet source into a :py:class:`~datafusion.dataframe.Dataframe`. This function will use the global context. Any functions or tables registered with another context may not be accessible when used with a DataFrame created using this function. Args: path: Path to the Parquet file. table_partition_cols: Partition columns. parquet_pruning: Whether the parquet reader should use the predicate to prune row groups. file_extension: File extension; only files with this extension are selected for data input. skip_metadata: Whether the parquet reader should skip any metadata that may be in the file schema. This can help avoid schema conflicts due to metadata. schema: An optional schema representing the parquet files. If None, the parquet reader will try to infer it based on data in the file. file_sort_order: Sort order for the file. Returns: DataFrame representation of the read Parquet files """ if table_partition_cols is None: table_partition_cols = [] return SessionContext.global_ctx().read_parquet( str(path), table_partition_cols, parquet_pruning, file_extension, skip_metadata, schema, file_sort_order, ) def read_json( path: str | pathlib.Path, schema: pa.Schema | None = None, schema_infer_max_records: int = 1000, file_extension: str = ".json", table_partition_cols: list[tuple[str, str | pa.DataType]] | None = None, file_compression_type: str | None = None, ) -> DataFrame: """Read a line-delimited JSON data source. This function will use the global context. Any functions or tables registered with another context may not be accessible when used with a DataFrame created using this function. Args: path: Path to the JSON file. schema: The data source schema. schema_infer_max_records: Maximum number of rows to read from JSON files for schema inference if needed. file_extension: File extension; only files with this extension are selected for data input. table_partition_cols: Partition columns. file_compression_type: File compression type. Returns: DataFrame representation of the read JSON files. """ if table_partition_cols is None: table_partition_cols = [] return SessionContext.global_ctx().read_json( str(path), schema, schema_infer_max_records, file_extension, table_partition_cols, file_compression_type, ) def read_csv( path: str | pathlib.Path | list[str] | list[pathlib.Path], schema: pa.Schema | None = None, has_header: bool = True, delimiter: str = ",", schema_infer_max_records: int = 1000, file_extension: str = ".csv", table_partition_cols: list[tuple[str, str | pa.DataType]] | None = None, file_compression_type: str | None = None, options: CsvReadOptions | None = None, ) -> DataFrame: """Read a CSV data source. This function will use the global context. Any functions or tables registered with another context may not be accessible when used with a DataFrame created using this function. Args: path: Path to the CSV file schema: An optional schema representing the CSV files. If None, the CSV reader will try to infer it based on data in file. has_header: Whether the CSV file have a header. If schema inference is run on a file with no headers, default column names are created. delimiter: An optional column delimiter. schema_infer_max_records: Maximum number of rows to read from CSV files for schema inference if needed. file_extension: File extension; only files with this extension are selected for data input. table_partition_cols: Partition columns. file_compression_type: File compression type. options: Set advanced options for CSV reading. This cannot be combined with any of the other options in this method. Returns: DataFrame representation of the read CSV files """ return SessionContext.global_ctx().read_csv( path, schema, has_header, delimiter, schema_infer_max_records, file_extension, table_partition_cols, file_compression_type, options, ) def read_avro( path: str | pathlib.Path, schema: pa.Schema | None = None, file_partition_cols: list[tuple[str, str | pa.DataType]] | None = None, file_extension: str = ".avro", ) -> DataFrame: """Create a :py:class:`DataFrame` for reading Avro data source. This function will use the global context. Any functions or tables registered with another context may not be accessible when used with a DataFrame created using this function. Args: path: Path to the Avro file. schema: The data source schema. file_partition_cols: Partition columns. file_extension: File extension to select. Returns: DataFrame representation of the read Avro file """ if file_partition_cols is None: file_partition_cols = [] return SessionContext.global_ctx().read_avro( str(path), schema, file_partition_cols, file_extension ) apache-datafusion-python-54.0.0/python/datafusion/ipc.py000066400000000000000000000220761521727353300233150ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Driver- and worker-side setup for distributing DataFusion expressions. When a :class:`Expr` is shipped to a worker process (e.g. through :func:`multiprocessing.Pool` or a Ray actor), the worker reconstructs the expression against a :class:`SessionContext`. If the expression references UDFs imported via the FFI capsule protocol — or any UDF the worker would otherwise resolve from its registered functions rather than from inside the shipped expression — install a configured :class:`SessionContext` once per worker: .. code-block:: python from datafusion import SessionContext from datafusion.ipc import set_worker_ctx def init_worker(): ctx = SessionContext() ctx.register_udaf(my_ffi_aggregate) set_worker_ctx(ctx) Built-in functions and Python UDFs (scalar, aggregate, window) travel inside the shipped expression itself and do not need pre-registration on the worker. .. note:: Serialization model Expressions containing Python UDFs (scalar, aggregate, window) are serialized using :mod:`cloudpickle`. The callable itself travels **by value** (bytecode and closure cells inlined), but any names the callable resolves via ``import`` are captured **by reference** and must be importable on the receiving worker. The serialized payload is stamped with the sender's Python ``(major, minor)`` version. Loading on a different minor version raises :class:`ValueError` with an actionable message — cloudpickle payloads are not portable across Python minor versions. See :meth:`datafusion.Expr.to_bytes` for examples of what travels by value vs. by reference. On the driver side, call :func:`set_sender_ctx` to control how :func:`pickle.dumps` encodes expressions — for example, to apply :meth:`SessionContext.with_python_udf_inlining` to every pickled expression on this thread: >>> import pickle >>> from datafusion import SessionContext, col, lit >>> from datafusion.ipc import clear_sender_ctx, set_sender_ctx >>> driver_ctx = SessionContext().with_python_udf_inlining(enabled=False) >>> set_sender_ctx(driver_ctx) >>> try: ... blob = pickle.dumps(col("a") + lit(1)) ... finally: ... clear_sender_ctx() >>> isinstance(blob, bytes) True Without a sender context the default codec is used (Python UDF inlining on). The sender context only affects pickle / ``to_bytes`` encoding; explicit ``expr.to_bytes(ctx)`` calls still use the supplied ``ctx``. The thread-local sender context holds a strong reference to the installed :class:`SessionContext` until :func:`clear_sender_ctx` is called or the thread exits. Long-running driver threads that install a sender context once and never clear it will retain that session for the lifetime of the thread; pair :func:`set_sender_ctx` with :func:`clear_sender_ctx` (e.g. in a ``try``/``finally``) when the sender context is only needed for a bounded scope. """ from __future__ import annotations import threading from typing import TYPE_CHECKING if TYPE_CHECKING: from datafusion.context import SessionContext __all__ = [ "clear_sender_ctx", "clear_worker_ctx", "get_sender_ctx", "get_worker_ctx", "set_sender_ctx", "set_worker_ctx", ] _local = threading.local() def set_worker_ctx(ctx: SessionContext) -> None: """Install this worker's :class:`SessionContext` for shipped expressions. Call once per worker — typically from a ``multiprocessing.Pool`` initializer or a Ray actor ``__init__``. Idempotent: overwrites any previous value. Stored in a thread-local slot, so each thread within a worker may install its own context independently. Examples: >>> from datafusion import SessionContext >>> from datafusion.ipc import set_worker_ctx, get_worker_ctx, clear_worker_ctx >>> set_worker_ctx(SessionContext()) >>> get_worker_ctx() is not None True >>> clear_worker_ctx() """ _local.ctx = ctx def clear_worker_ctx() -> None: """Remove this worker's installed :class:`SessionContext`. After clearing, expressions reconstructed in this worker fall back to the global :class:`SessionContext` — adequate for built-ins and Python UDFs (scalar, aggregate, window), but anything imported via the FFI capsule protocol must be registered on the global context to resolve. Examples: >>> from datafusion import SessionContext >>> from datafusion.ipc import set_worker_ctx, clear_worker_ctx, get_worker_ctx >>> set_worker_ctx(SessionContext()) >>> clear_worker_ctx() >>> get_worker_ctx() is None True """ if hasattr(_local, "ctx"): del _local.ctx def get_worker_ctx() -> SessionContext | None: """Return this worker's installed :class:`SessionContext`, or ``None``. Examples: >>> from datafusion.ipc import get_worker_ctx, clear_worker_ctx >>> clear_worker_ctx() >>> get_worker_ctx() is None True """ return getattr(_local, "ctx", None) def set_sender_ctx(ctx: SessionContext) -> None: """Install this driver's :class:`SessionContext` for outbound pickles. Controls how :func:`pickle.dumps` encodes :class:`Expr` instances on this thread. The most useful application is propagating a session configured with :meth:`SessionContext.with_python_udf_inlining` so the toggle takes effect through pickle (which otherwise calls :meth:`Expr.to_bytes` with no context and uses the default codec). Idempotent: overwrites any previous value. Stored in a thread-local slot, so worker threads on the driver may install their own contexts. Does not affect :meth:`Expr.to_bytes` calls that pass an explicit ``ctx`` — those continue to use the supplied context. Examples: >>> from datafusion import SessionContext >>> from datafusion.ipc import set_sender_ctx, get_sender_ctx >>> driver = SessionContext().with_python_udf_inlining(enabled=False) >>> set_sender_ctx(driver) >>> get_sender_ctx() is driver True """ _local.sender_ctx = ctx def clear_sender_ctx() -> None: """Remove this driver's installed sender :class:`SessionContext`. After clearing, pickled expressions fall back to the default codec (Python UDF inlining on). Examples: >>> from datafusion import SessionContext >>> from datafusion.ipc import ( ... set_sender_ctx, clear_sender_ctx, get_sender_ctx, ... ) >>> set_sender_ctx(SessionContext()) >>> clear_sender_ctx() >>> get_sender_ctx() is None True """ if hasattr(_local, "sender_ctx"): del _local.sender_ctx def get_sender_ctx() -> SessionContext | None: """Return this driver's installed sender :class:`SessionContext`, or ``None``. Examples: >>> from datafusion.ipc import get_sender_ctx, clear_sender_ctx >>> clear_sender_ctx() >>> get_sender_ctx() is None True """ return getattr(_local, "sender_ctx", None) def _resolve_ctx( explicit_ctx: SessionContext | None = None, ) -> SessionContext: """Resolve a context for Expr reconstruction. Priority: explicit argument > worker context > global context. Falling back to the global :class:`SessionContext` (instead of a freshly constructed one) preserves any registrations the user has installed on it. Examples: >>> from datafusion import SessionContext >>> from datafusion.ipc import _resolve_ctx, clear_worker_ctx >>> clear_worker_ctx() >>> isinstance(_resolve_ctx(), SessionContext) True >>> ctx = SessionContext() >>> _resolve_ctx(ctx) is ctx True """ if explicit_ctx is not None: return explicit_ctx worker = get_worker_ctx() if worker is not None: return worker # Lazy import: `datafusion/__init__.py` imports `datafusion.ipc` # before `datafusion.context`, so a module-top import would force # `datafusion.context` to load mid-init of `datafusion.ipc`. The # cycle is benign today (context.py only pulls expr.py at module # scope, neither pulls ipc.py back), but a single new import in # context.py's transitive deps could turn it into a real cycle. # Deferring keeps `datafusion.ipc` import-order-independent. from datafusion.context import SessionContext # noqa: PLC0415 return SessionContext.global_ctx() apache-datafusion-python-54.0.0/python/datafusion/object_store.py000066400000000000000000000021521521727353300252150ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Object store functionality.""" from ._internal import object_store AmazonS3 = object_store.AmazonS3 GoogleCloud = object_store.GoogleCloud LocalFileSystem = object_store.LocalFileSystem MicrosoftAzure = object_store.MicrosoftAzure Http = object_store.Http __all__ = ["AmazonS3", "GoogleCloud", "Http", "LocalFileSystem", "MicrosoftAzure"] apache-datafusion-python-54.0.0/python/datafusion/options.py000066400000000000000000000257451521727353300242430ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Options for reading various file formats.""" from __future__ import annotations import warnings from typing import TYPE_CHECKING import pyarrow as pa from datafusion.expr import sort_list_to_raw_sort_list if TYPE_CHECKING: from datafusion.expr import SortExpr from ._internal import options __all__ = ["CsvReadOptions"] DEFAULT_MAX_INFER_SCHEMA = 1000 class CsvReadOptions: """Options for reading CSV files. This class provides a builder pattern for configuring CSV reading options. All methods starting with ``with_`` return ``self`` to allow method chaining. """ def __init__( self, *, has_header: bool = True, delimiter: str = ",", quote: str = '"', terminator: str | None = None, escape: str | None = None, comment: str | None = None, newlines_in_values: bool = False, schema: pa.Schema | None = None, schema_infer_max_records: int = DEFAULT_MAX_INFER_SCHEMA, file_extension: str = ".csv", table_partition_cols: list[tuple[str, pa.DataType]] | None = None, file_compression_type: str = "", file_sort_order: list[list[SortExpr]] | None = None, null_regex: str | None = None, truncated_rows: bool = False, ) -> None: """Initialize CsvReadOptions. Args: has_header: Does the CSV file have a header row? If schema inference is run on a file with no headers, default column names are created. delimiter: Column delimiter character. Must be a single ASCII character. quote: Quote character for fields containing delimiters or newlines. Must be a single ASCII character. terminator: Optional line terminator character. If ``None``, uses CRLF. Must be a single ASCII character. escape: Optional escape character for quotes. Must be a single ASCII character. comment: If specified, lines beginning with this character are ignored. Must be a single ASCII character. newlines_in_values: Whether newlines in quoted values are supported. Parsing newlines in quoted values may be affected by execution behavior such as parallel file scanning. Setting this to ``True`` ensures that newlines in values are parsed successfully, which may reduce performance. schema: Optional PyArrow schema representing the CSV files. If ``None``, the CSV reader will try to infer it based on data in the file. schema_infer_max_records: Maximum number of rows to read from CSV files for schema inference if needed. file_extension: File extension; only files with this extension are selected for data input. table_partition_cols: Partition columns as a list of tuples of (column_name, data_type). file_compression_type: File compression type. Supported values are ``"gzip"``, ``"bz2"``, ``"xz"``, ``"zstd"``, or empty string for uncompressed. file_sort_order: Optional sort order of the files as a list of sort expressions per file. null_regex: Optional regex pattern to match null values in the CSV. truncated_rows: Whether to allow truncated rows when parsing. By default this is ``False`` and will error if the CSV rows have different lengths. When set to ``True``, it will allow records with less than the expected number of columns and fill the missing columns with nulls. If the record's schema is not nullable, it will still return an error. """ validate_single_character("delimiter", delimiter) validate_single_character("quote", quote) validate_single_character("terminator", terminator) validate_single_character("escape", escape) validate_single_character("comment", comment) self.has_header = has_header self.delimiter = delimiter self.quote = quote self.terminator = terminator self.escape = escape self.comment = comment self.newlines_in_values = newlines_in_values self.schema = schema self.schema_infer_max_records = schema_infer_max_records self.file_extension = file_extension self.table_partition_cols = table_partition_cols or [] self.file_compression_type = file_compression_type self.file_sort_order = file_sort_order or [] self.null_regex = null_regex self.truncated_rows = truncated_rows def with_has_header(self, has_header: bool) -> CsvReadOptions: """Configure whether the CSV has a header row.""" self.has_header = has_header return self def with_delimiter(self, delimiter: str) -> CsvReadOptions: """Configure the column delimiter.""" self.delimiter = delimiter return self def with_quote(self, quote: str) -> CsvReadOptions: """Configure the quote character.""" self.quote = quote return self def with_terminator(self, terminator: str | None) -> CsvReadOptions: """Configure the line terminator character.""" self.terminator = terminator return self def with_escape(self, escape: str | None) -> CsvReadOptions: """Configure the escape character.""" self.escape = escape return self def with_comment(self, comment: str | None) -> CsvReadOptions: """Configure the comment character.""" self.comment = comment return self def with_newlines_in_values(self, newlines_in_values: bool) -> CsvReadOptions: """Configure whether newlines in values are supported.""" self.newlines_in_values = newlines_in_values return self def with_schema(self, schema: pa.Schema | None) -> CsvReadOptions: """Configure the schema.""" self.schema = schema return self def with_schema_infer_max_records( self, schema_infer_max_records: int ) -> CsvReadOptions: """Configure maximum records for schema inference.""" self.schema_infer_max_records = schema_infer_max_records return self def with_file_extension(self, file_extension: str) -> CsvReadOptions: """Configure the file extension filter.""" self.file_extension = file_extension return self def with_table_partition_cols( self, table_partition_cols: list[tuple[str, pa.DataType]] ) -> CsvReadOptions: """Configure table partition columns.""" self.table_partition_cols = table_partition_cols return self def with_file_compression_type(self, file_compression_type: str) -> CsvReadOptions: """Configure file compression type.""" self.file_compression_type = file_compression_type return self def with_file_sort_order( self, file_sort_order: list[list[SortExpr]] ) -> CsvReadOptions: """Configure file sort order.""" self.file_sort_order = file_sort_order return self def with_null_regex(self, null_regex: str | None) -> CsvReadOptions: """Configure null value regex pattern.""" self.null_regex = null_regex return self def with_truncated_rows(self, truncated_rows: bool) -> CsvReadOptions: """Configure whether to allow truncated rows.""" self.truncated_rows = truncated_rows return self def to_inner(self) -> options.CsvReadOptions: """Convert this object into the underlying Rust structure. This is intended for internal use only. """ file_sort_order = ( [] if self.file_sort_order is None else [ sort_list_to_raw_sort_list(sort_list) for sort_list in self.file_sort_order ] ) return options.CsvReadOptions( has_header=self.has_header, delimiter=ord(self.delimiter[0]) if self.delimiter else ord(","), quote=ord(self.quote[0]) if self.quote else ord('"'), terminator=ord(self.terminator[0]) if self.terminator else None, escape=ord(self.escape[0]) if self.escape else None, comment=ord(self.comment[0]) if self.comment else None, newlines_in_values=self.newlines_in_values, schema=self.schema, schema_infer_max_records=self.schema_infer_max_records, file_extension=self.file_extension, table_partition_cols=_convert_table_partition_cols( self.table_partition_cols ), file_compression_type=self.file_compression_type or "", file_sort_order=file_sort_order, null_regex=self.null_regex, truncated_rows=self.truncated_rows, ) def validate_single_character(name: str, value: str | None) -> None: if value is not None and len(value) != 1: message = f"{name} must be a single character" raise ValueError(message) def _convert_table_partition_cols( table_partition_cols: list[tuple[str, str | pa.DataType]], ) -> list[tuple[str, pa.DataType]]: warn = False converted_table_partition_cols = [] for col, data_type in table_partition_cols: if isinstance(data_type, str): warn = True if data_type == "string": converted_data_type = pa.string() elif data_type == "int": converted_data_type = pa.int32() else: message = ( f"Unsupported literal data type '{data_type}' for partition " "column. Supported types are 'string' and 'int'" ) raise ValueError(message) else: converted_data_type = data_type converted_table_partition_cols.append((col, converted_data_type)) if warn: message = ( "using literals for table_partition_cols data types is deprecated," "use pyarrow types instead" ) warnings.warn( message, category=DeprecationWarning, stacklevel=2, ) return converted_table_partition_cols apache-datafusion-python-54.0.0/python/datafusion/plan.py000066400000000000000000000336661521727353300235030ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """This module supports physical and logical plans in DataFusion.""" from __future__ import annotations import warnings from typing import TYPE_CHECKING, Any import datafusion._internal as df_internal if TYPE_CHECKING: import datetime from datafusion.context import SessionContext __all__ = [ "ExecutionPlan", "LogicalPlan", "Metric", "MetricsSet", ] class LogicalPlan: # noqa: PLW1641 """Logical Plan. A `LogicalPlan` is a node in a tree of relational operators (such as Projection or Filter). Represents transforming an input relation (table) to an output relation (table) with a potentially different schema. Plans form a dataflow tree where data flows from leaves up to the root to produce the query result. A `LogicalPlan` can be created by the SQL query planner, the DataFrame API, or programmatically (for example custom query languages). """ def __init__(self, plan: df_internal.LogicalPlan) -> None: """This constructor should not be called by the end user.""" self._raw_plan = plan def to_variant(self) -> Any: """Convert the logical plan into its specific variant.""" return self._raw_plan.to_variant() def inputs(self) -> list[LogicalPlan]: """Returns the list of inputs to the logical plan.""" return [LogicalPlan(p) for p in self._raw_plan.inputs()] def __repr__(self) -> str: """Generate a printable representation of the plan.""" return self._raw_plan.__repr__() def display(self) -> str: """Print the logical plan.""" return self._raw_plan.display() def display_indent(self) -> str: """Print an indented form of the logical plan.""" return self._raw_plan.display_indent() def display_indent_schema(self) -> str: """Print an indented form of the schema for the logical plan.""" return self._raw_plan.display_indent_schema() def display_graphviz(self) -> str: """Print the graph visualization of the logical plan. Returns a `format`able structure that produces lines meant for graphical display using the `DOT` language. This format can be visualized using software from [`graphviz`](https://graphviz.org/) """ return self._raw_plan.display_graphviz() @staticmethod def from_bytes(ctx: SessionContext, data: bytes) -> LogicalPlan: """Create a LogicalPlan from serialized protobuf bytes. Decoding routes through the session's installed `LogicalExtensionCodec`. Tables created in memory from record batches are currently not supported. """ return LogicalPlan(df_internal.LogicalPlan.from_bytes(ctx.ctx, data)) def to_bytes(self, ctx: SessionContext | None = None) -> bytes: """Convert a LogicalPlan to serialized protobuf bytes. When ``ctx`` is supplied, encoding routes through the session's installed `LogicalExtensionCodec` so user FFI codecs (registered via :py:meth:`SessionContext.with_logical_extension_codec`) see the encode path. With ``ctx=None`` a default codec is used. Tables created in memory from record batches are currently not supported. """ ctx_arg = ctx.ctx if ctx is not None else None return self._raw_plan.to_bytes(ctx_arg) @staticmethod def from_proto(ctx: SessionContext, data: bytes) -> LogicalPlan: """Deprecated alias for :meth:`from_bytes`.""" warnings.warn( "LogicalPlan.from_proto is deprecated; use from_bytes instead", DeprecationWarning, stacklevel=2, ) return LogicalPlan.from_bytes(ctx, data) def to_proto(self) -> bytes: """Deprecated alias for :meth:`to_bytes`.""" warnings.warn( "LogicalPlan.to_proto is deprecated; use to_bytes instead", DeprecationWarning, stacklevel=2, ) return self.to_bytes() def __eq__(self, other: LogicalPlan) -> bool: """Test equality.""" if not isinstance(other, LogicalPlan): return False return self._raw_plan.__eq__(other._raw_plan) class ExecutionPlan: """Represent nodes in the DataFusion Physical Plan.""" def __init__(self, plan: df_internal.ExecutionPlan) -> None: """This constructor should not be called by the end user.""" self._raw_plan = plan def children(self) -> list[ExecutionPlan]: """Get a list of children `ExecutionPlan` that act as inputs to this plan. The returned list will be empty for leaf nodes such as scans, will contain a single value for unary nodes, or two values for binary nodes (such as joins). """ return [ExecutionPlan(e) for e in self._raw_plan.children()] def display(self) -> str: """Print the physical plan.""" return self._raw_plan.display() def display_indent(self) -> str: """Print an indented form of the physical plan.""" return self._raw_plan.display_indent() def __repr__(self) -> str: """Print a string representation of the physical plan.""" return self._raw_plan.__repr__() @property def partition_count(self) -> int: """Returns the number of partitions in the physical plan.""" return self._raw_plan.partition_count @staticmethod def from_bytes(ctx: SessionContext, data: bytes) -> ExecutionPlan: """Create an ExecutionPlan from serialized protobuf bytes. Decoding routes through the session's installed `PhysicalExtensionCodec`. Tables created in memory from record batches are currently not supported. """ return ExecutionPlan(df_internal.ExecutionPlan.from_bytes(ctx.ctx, data)) def to_bytes(self, ctx: SessionContext | None = None) -> bytes: """Convert an ExecutionPlan into serialized protobuf bytes. When ``ctx`` is supplied, encoding routes through the session's installed `PhysicalExtensionCodec`. Tables created in memory from record batches are currently not supported. """ ctx_arg = ctx.ctx if ctx is not None else None return self._raw_plan.to_bytes(ctx_arg) @staticmethod def from_proto(ctx: SessionContext, data: bytes) -> ExecutionPlan: """Deprecated alias for :meth:`from_bytes`.""" warnings.warn( "ExecutionPlan.from_proto is deprecated; use from_bytes instead", DeprecationWarning, stacklevel=2, ) return ExecutionPlan.from_bytes(ctx, data) def to_proto(self) -> bytes: """Deprecated alias for :meth:`to_bytes`.""" warnings.warn( "ExecutionPlan.to_proto is deprecated; use to_bytes instead", DeprecationWarning, stacklevel=2, ) return self.to_bytes() def metrics(self) -> MetricsSet | None: """Return metrics for this plan node, or None if this plan has no MetricsSet. Some operators (e.g. DataSourceExec) eagerly initialize a MetricsSet when the plan is created, so this may return a set even before execution. Metric *values* (such as ``output_rows``) are only meaningful after the DataFrame has been executed. """ raw = self._raw_plan.metrics() if raw is None: return None return MetricsSet(raw) def collect_metrics(self) -> list[tuple[str, MetricsSet]]: """Return runtime statistics for each step of the query execution. DataFusion executes a query as a pipeline of operators — for example a data source scan, followed by a filter, followed by a projection. After the DataFrame has been executed (via :py:meth:`~datafusion.DataFrame.collect`, :py:meth:`~datafusion.DataFrame.execute_stream`, etc.), each operator records statistics such as how many rows it produced and how much CPU time it consumed. Each entry in the returned list corresponds to one operator that recorded metrics. The first element of the tuple is the operator's description string — the same text shown by :py:meth:`display_indent` — which identifies both the operator type and its key parameters, for example ``"FilterExec: column1@0 > 1"`` or ``"DataSourceExec: partitions=1"``. Returns: A list of ``(description, MetricsSet)`` tuples ordered from the outermost operator (top of the execution tree) down to the data-source leaves. Only operators that recorded at least one metric are included. Returns an empty list if called before the DataFrame has been executed. """ result: list[tuple[str, MetricsSet]] = [] def _walk(node: ExecutionPlan) -> None: ms = node.metrics() if ms is not None: result.append((node.display(), ms)) for child in node.children(): _walk(child) _walk(self) return result class MetricsSet: """A set of metrics for a single execution plan operator. A physical plan operator runs independently across one or more partitions. :py:meth:`metrics` returns the raw per-partition :py:class:`Metric` objects. The convenience properties (:py:attr:`output_rows`, :py:attr:`elapsed_compute`, etc.) automatically sum the named metric across *all* partitions, giving a single aggregate value for the operator as a whole. """ def __init__(self, raw: df_internal.MetricsSet) -> None: """This constructor should not be called by the end user.""" self._raw = raw def metrics(self) -> list[Metric]: """Return all individual metrics in this set.""" return [Metric(m) for m in self._raw.metrics()] @property def output_rows(self) -> int | None: """Sum of output_rows across all partitions.""" return self._raw.output_rows() @property def elapsed_compute(self) -> int | None: """Total CPU time (in nanoseconds) spent inside this operator's execute loop. Summed across all partitions. Returns ``None`` if no ``elapsed_compute`` metric was recorded. """ return self._raw.elapsed_compute() @property def spill_count(self) -> int | None: """Number of times this operator spilled data to disk due to memory pressure. This is a count of spill events, not a byte count. Summed across all partitions. Returns ``None`` if no ``spill_count`` metric was recorded. """ return self._raw.spill_count() @property def spilled_bytes(self) -> int | None: """Sum of spilled_bytes across all partitions.""" return self._raw.spilled_bytes() @property def spilled_rows(self) -> int | None: """Sum of spilled_rows across all partitions.""" return self._raw.spilled_rows() def sum_by_name(self, name: str) -> int | None: """Sum the named metric across all partitions. Useful for accessing any metric not exposed as a first-class property. Returns ``None`` if no metric with the given name was recorded. Args: name: The metric name, e.g. ``"output_rows"`` or ``"elapsed_compute"``. """ return self._raw.sum_by_name(name) def __repr__(self) -> str: """Return a string representation of the metrics set.""" return repr(self._raw) class Metric: """A single execution metric with name, value, partition, and labels.""" def __init__(self, raw: df_internal.Metric) -> None: """This constructor should not be called by the end user.""" self._raw = raw @property def name(self) -> str: """The name of this metric (e.g. ``output_rows``).""" return self._raw.name @property def value(self) -> int | datetime.datetime | None: """The value of this metric. Returns an ``int`` for counters, gauges, and time-based metrics (nanoseconds), a :py:class:`~datetime.datetime` (UTC) for ``start_timestamp`` / ``end_timestamp`` metrics, or ``None`` when the value has not been set or is not representable. """ return self._raw.value @property def value_as_datetime(self) -> datetime.datetime | None: """The value as a UTC :py:class:`~datetime.datetime` for timestamp metrics. Returns ``None`` for all non-timestamp metrics and for timestamp metrics whose value has not been set (e.g. before execution). """ return self._raw.value_as_datetime @property def partition(self) -> int | None: """The 0-based partition index this metric applies to. Returns ``None`` for metrics that are not partition-specific (i.e. they apply globally across all partitions of the operator). """ return self._raw.partition def labels(self) -> dict[str, str]: """Return the labels associated with this metric. Labels provide additional context for a metric. For example:: metric.labels() # {'output_type': 'final'} """ return self._raw.labels() def __repr__(self) -> str: """Return a string representation of the metric.""" return repr(self._raw) apache-datafusion-python-54.0.0/python/datafusion/py.typed000066400000000000000000000014201521727353300236550ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License.apache-datafusion-python-54.0.0/python/datafusion/record_batch.py000066400000000000000000000071221521727353300251540ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """This module provides the classes for handling record batches. These are typically the result of dataframe :py:func:`datafusion.dataframe.execute_stream` operations. """ from __future__ import annotations from typing import TYPE_CHECKING if TYPE_CHECKING: import pyarrow as pa import typing_extensions import datafusion._internal as df_internal class RecordBatch: """This class is essentially a wrapper for :py:class:`pa.RecordBatch`.""" def __init__(self, record_batch: df_internal.RecordBatch) -> None: """This constructor is generally not called by the end user. See the :py:class:`RecordBatchStream` iterator for generating this class. """ self.record_batch = record_batch def to_pyarrow(self) -> pa.RecordBatch: """Convert to :py:class:`pa.RecordBatch`.""" return self.record_batch.to_pyarrow() def __arrow_c_array__( self, requested_schema: object | None = None ) -> tuple[object, object]: """Export the record batch via the Arrow C Data Interface. This allows zero-copy interchange with libraries that support the `Arrow PyCapsule interface `_. Args: requested_schema: Attempt to provide the record batch using this schema. Only straightforward projections such as column selection or reordering are applied. Returns: Two Arrow PyCapsule objects representing the ``ArrowArray`` and ``ArrowSchema``. """ return self.record_batch.__arrow_c_array__(requested_schema) class RecordBatchStream: """This class represents a stream of record batches. These are typically the result of a :py:func:`~datafusion.dataframe.DataFrame.execute_stream` operation. """ def __init__(self, record_batch_stream: df_internal.RecordBatchStream) -> None: """This constructor is typically not called by the end user.""" self.rbs = record_batch_stream def next(self) -> RecordBatch: """See :py:func:`__next__` for the iterator function.""" return next(self) async def __anext__(self) -> RecordBatch: """Return the next :py:class:`RecordBatch` in the stream asynchronously.""" next_batch = await self.rbs.__anext__() return RecordBatch(next_batch) def __next__(self) -> RecordBatch: """Return the next :py:class:`RecordBatch` in the stream.""" next_batch = next(self.rbs) return RecordBatch(next_batch) def __aiter__(self) -> typing_extensions.Self: """Return an asynchronous iterator over record batches.""" return self def __iter__(self) -> typing_extensions.Self: """Return an iterator over record batches.""" return self apache-datafusion-python-54.0.0/python/datafusion/substrait.py000066400000000000000000000124511521727353300245560ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """This module provides support for using substrait with datafusion. For additional information about substrait, see https://substrait.io/ for more information about substrait. """ from __future__ import annotations from typing import TYPE_CHECKING from datafusion.plan import LogicalPlan from ._internal import substrait as substrait_internal if TYPE_CHECKING: import pathlib from datafusion.context import SessionContext __all__ = [ "Consumer", "Plan", "Producer", "Serde", ] class Plan: """A class representing an encodable substrait plan.""" def __init__(self, plan: substrait_internal.Plan) -> None: """Create a substrait plan. The user should not have to call this constructor directly. Rather, it should be created via :py:class:`Serde` or py:class:`Producer` classes in this module. """ self.plan_internal = plan def encode(self) -> bytes: """Encode the plan to bytes. Returns: Encoded plan. """ return self.plan_internal.encode() def to_json(self) -> str: """Get the JSON representation of the Substrait plan. Returns: A JSON representation of the Substrait plan. """ return self.plan_internal.to_json() @staticmethod def from_json(json: str) -> Plan: """Parse a plan from a JSON string representation. Args: json: JSON representation of a Substrait plan. Returns: Plan object representing the Substrait plan. """ return Plan(substrait_internal.Plan.from_json(json)) class Serde: """Provides the ``Substrait`` serialization and deserialization.""" @staticmethod def serialize(sql: str, ctx: SessionContext, path: str | pathlib.Path) -> None: """Serialize a SQL query to a Substrait plan and write it to a file. Args: sql:SQL query to serialize. ctx: SessionContext to use. path: Path to write the Substrait plan to. """ return substrait_internal.Serde.serialize(sql, ctx.ctx, str(path)) @staticmethod def serialize_to_plan(sql: str, ctx: SessionContext) -> Plan: """Serialize a SQL query to a Substrait plan. Args: sql: SQL query to serialize. ctx: SessionContext to use. Returns: Substrait plan. """ return Plan(substrait_internal.Serde.serialize_to_plan(sql, ctx.ctx)) @staticmethod def serialize_bytes(sql: str, ctx: SessionContext) -> bytes: """Serialize a SQL query to a Substrait plan as bytes. Args: sql: SQL query to serialize. ctx: SessionContext to use. Returns: Substrait plan as bytes. """ return substrait_internal.Serde.serialize_bytes(sql, ctx.ctx) @staticmethod def deserialize(path: str | pathlib.Path) -> Plan: """Deserialize a Substrait plan from a file. Args: path: Path to read the Substrait plan from. Returns: Substrait plan. """ return Plan(substrait_internal.Serde.deserialize(str(path))) @staticmethod def deserialize_bytes(proto_bytes: bytes) -> Plan: """Deserialize a Substrait plan from bytes. Args: proto_bytes: Bytes to read the Substrait plan from. Returns: Substrait plan. """ return Plan(substrait_internal.Serde.deserialize_bytes(proto_bytes)) class Producer: """Generates substrait plans from a logical plan.""" @staticmethod def to_substrait_plan(logical_plan: LogicalPlan, ctx: SessionContext) -> Plan: """Convert a DataFusion LogicalPlan to a Substrait plan. Args: logical_plan: LogicalPlan to convert. ctx: SessionContext to use. Returns: Substrait plan. """ return Plan( substrait_internal.Producer.to_substrait_plan( logical_plan._raw_plan, ctx.ctx ) ) class Consumer: """Generates a logical plan from a substrait plan.""" @staticmethod def from_substrait_plan(ctx: SessionContext, plan: Plan) -> LogicalPlan: """Convert a Substrait plan to a DataFusion LogicalPlan. Args: ctx: SessionContext to use. plan: Substrait plan to convert. Returns: LogicalPlan. """ return LogicalPlan( substrait_internal.Consumer.from_substrait_plan(ctx.ctx, plan.plan_internal) ) apache-datafusion-python-54.0.0/python/datafusion/unparser.py000066400000000000000000000051701521727353300243750ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """This module provides support for unparsing datafusion plans to SQL. For additional information about unparsing, see https://docs.rs/datafusion-sql/latest/datafusion_sql/unparser/index.html """ from ._internal import unparser as unparser_internal from .plan import LogicalPlan class Dialect: """DataFusion data catalog.""" def __init__(self, dialect: unparser_internal.Dialect) -> None: """This constructor is not typically called by the end user.""" self.dialect = dialect @staticmethod def default() -> "Dialect": """Create a new default dialect.""" return Dialect(unparser_internal.Dialect.default()) @staticmethod def mysql() -> "Dialect": """Create a new MySQL dialect.""" return Dialect(unparser_internal.Dialect.mysql()) @staticmethod def postgres() -> "Dialect": """Create a new PostgreSQL dialect.""" return Dialect(unparser_internal.Dialect.postgres()) @staticmethod def sqlite() -> "Dialect": """Create a new SQLite dialect.""" return Dialect(unparser_internal.Dialect.sqlite()) @staticmethod def duckdb() -> "Dialect": """Create a new DuckDB dialect.""" return Dialect(unparser_internal.Dialect.duckdb()) class Unparser: """DataFusion unparser.""" def __init__(self, dialect: Dialect) -> None: """This constructor is not typically called by the end user.""" self.unparser = unparser_internal.Unparser(dialect.dialect) def plan_to_sql(self, plan: LogicalPlan) -> str: """Convert a logical plan to a SQL string.""" return self.unparser.plan_to_sql(plan._raw_plan) def with_pretty(self, pretty: bool) -> "Unparser": """Set the pretty flag.""" self.unparser = self.unparser.with_pretty(pretty) return self __all__ = [ "Dialect", "Unparser", ] apache-datafusion-python-54.0.0/python/datafusion/user_defined.py000066400000000000000000001343611521727353300251770ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. """Provides the user-defined functions for evaluation of dataframes.""" from __future__ import annotations import functools from abc import ABCMeta, abstractmethod from enum import Enum from typing import TYPE_CHECKING, Any, Protocol, TypeGuard, TypeVar, cast, overload import pyarrow as pa import datafusion._internal as df_internal from datafusion import SessionContext from datafusion.expr import Expr if TYPE_CHECKING: from _typeshed import CapsuleType as _PyCapsule _R = TypeVar("_R", bound=pa.Array) from collections.abc import Callable, Sequence class Volatility(Enum): """Defines how stable or volatile a function is. When setting the volatility of a function, you can either pass this enumeration or a ``str``. The ``str`` equivalent is the lower case value of the name (`"immutable"`, `"stable"`, or `"volatile"`). """ Immutable = 1 """An immutable function will always return the same output when given the same input. DataFusion will attempt to inline immutable functions during planning. """ Stable = 2 """ Returns the same value for a given input within a single queries. A stable function may return different values given the same input across different queries but must return the same value for a given input within a query. An example of this is the ``Now`` function. DataFusion will attempt to inline ``Stable`` functions during planning, when possible. For query ``select col1, now() from t1``, it might take a while to execute but ``now()`` column will be the same for each output row, which is evaluated during planning. """ Volatile = 3 """A volatile function may change the return value from evaluation to evaluation. Multiple invocations of a volatile function may return different results when used in the same query. An example of this is the random() function. DataFusion can not evaluate such functions during planning. In the query ``select col1, random() from t1``, ``random()`` function will be evaluated for each output row, resulting in a unique random value for each row. """ def __str__(self) -> str: """Returns the string equivalent.""" return self.name.lower() def data_type_or_field_to_field(value: pa.DataType | pa.Field, name: str) -> pa.Field: """Helper function to return a Field from either a Field or DataType.""" if isinstance(value, pa.Field): return value return pa.field(name, type=value) def data_types_or_fields_to_field_list( inputs: Sequence[pa.Field | pa.DataType] | pa.Field | pa.DataType, ) -> list[pa.Field]: """Helper function to return a list of Fields.""" if isinstance(inputs, pa.DataType): return [pa.field("value", type=inputs)] if isinstance(inputs, pa.Field): return [inputs] return [ data_type_or_field_to_field(v, f"value_{idx}") for (idx, v) in enumerate(inputs) ] class ScalarUDFExportable(Protocol): """Type hint for object that has __datafusion_scalar_udf__ PyCapsule.""" def __datafusion_scalar_udf__(self) -> object: ... # noqa: D105 def _is_pycapsule(value: object) -> TypeGuard[_PyCapsule]: """Return ``True`` when ``value`` is a CPython ``PyCapsule``.""" return value.__class__.__name__ == "PyCapsule" class LogicalExtensionCodecExportable(Protocol): """Type hint for objects exposing ``__datafusion_logical_extension_codec__``.""" def __datafusion_logical_extension_codec__(self) -> object: ... # noqa: D105 class PhysicalExtensionCodecExportable(Protocol): """Type hint for objects exposing ``__datafusion_physical_extension_codec__``.""" def __datafusion_physical_extension_codec__(self) -> object: ... # noqa: D105 class ScalarUDF: """Class for performing scalar user-defined functions (UDF). Scalar UDFs operate on a row by row basis. See also :py:class:`AggregateUDF` for operating on a group of rows. """ def __init__( self, name: str, func: Callable[..., _R], input_fields: list[pa.Field], return_field: pa.Field, volatility: Volatility | str, ) -> None: """Instantiate a scalar user-defined function (UDF). See helper method :py:func:`udf` for argument details. """ if hasattr(func, "__datafusion_scalar_udf__"): self._udf = df_internal.ScalarUDF.from_pycapsule(func) return if isinstance(input_fields, pa.DataType): input_fields = [input_fields] self._udf = df_internal.ScalarUDF( name, func, input_fields, return_field, str(volatility) ) @classmethod def _from_internal(cls, internal: df_internal.ScalarUDF) -> ScalarUDF: """Wrap an already-constructed internal ``ScalarUDF`` handle. Used by :py:meth:`SessionContext.udf` to surface a function looked up from the session's function registry without re-running :py:meth:`__init__`. """ wrapper = cls.__new__(cls) wrapper._udf = internal return wrapper @property def name(self) -> str: """Return the registered name of this UDF. For UDFs imported via the FFI capsule protocol, this is the name the capsule itself reports — not the ``name`` argument passed to the constructor (which is ignored on the FFI path). Examples: >>> import pyarrow as pa >>> from datafusion import udf >>> double = udf( ... lambda arr: pa.array([(v.as_py() or 0) * 2 for v in arr]), ... [pa.int64()], ... pa.int64(), ... volatility="immutable", ... name="double", ... ) >>> double.name 'double' """ return self._udf.name def __repr__(self) -> str: """Print a string representation of the Scalar UDF.""" return self._udf.__repr__() def __call__(self, *args: Expr) -> Expr: """Execute the UDF. This function is not typically called by an end user. These calls will occur during the evaluation of the dataframe. """ args_raw = [arg.expr for arg in args] return Expr(self._udf.__call__(*args_raw)) @overload @staticmethod def udf( input_fields: Sequence[pa.DataType | pa.Field] | pa.DataType | pa.Field, return_field: pa.DataType | pa.Field, volatility: Volatility | str, name: str | None = None, ) -> Callable[..., ScalarUDF]: ... @overload @staticmethod def udf( func: Callable[..., _R], input_fields: Sequence[pa.DataType | pa.Field] | pa.DataType | pa.Field, return_field: pa.DataType | pa.Field, volatility: Volatility | str, name: str | None = None, ) -> ScalarUDF: ... @overload @staticmethod def udf(func: ScalarUDFExportable) -> ScalarUDF: ... @staticmethod def udf(*args: Any, **kwargs: Any): # noqa: D417 """Create a new User-Defined Function (UDF). This class can be used both as either a function or a decorator. Usage: - As a function: ``udf(func, input_fields, return_field, volatility, name)``. - As a decorator: ``@udf(input_fields, return_field, volatility, name)``. When used a decorator, do **not** pass ``func`` explicitly. In lieu of passing a PyArrow Field, you can pass a DataType for simplicity. When you do so, it will be assumed that the nullability of the inputs and output are True and that they have no metadata. Args: func (Callable, optional): Only needed when calling as a function. Skip this argument when using `udf` as a decorator. If you have a Rust backed ScalarUDF within a PyCapsule, you can pass this parameter and ignore the rest. They will be determined directly from the underlying function. See the online documentation for more information. input_fields (list[pa.Field | pa.DataType]): The data types or Fields of the arguments to ``func``. This list must be of the same length as the number of arguments. return_field (_R): The field of the return value from the function. volatility (Volatility | str): See `Volatility` for allowed values. name (Optional[str]): A descriptive name for the function. Returns: A user-defined function that can be used in SQL expressions, data aggregation, or window function calls. Examples: Using ``udf`` as a function: >>> import pyarrow.compute as pc >>> from datafusion.user_defined import ScalarUDF >>> def double_func(x): ... return pc.multiply(x, 2) >>> double_udf = ScalarUDF.udf( ... double_func, [pa.int64()], pa.int64(), ... "volatile", "double_it") Using ``udf`` as a decorator: >>> @ScalarUDF.udf([pa.int64()], pa.int64(), "volatile") ... def decorator_double_udf(x): ... return pc.multiply(x, 3) Apply to a dataframe: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"x": [1, 2, 3]}) >>> df.select(double_udf(col("x")).alias("result")).to_pydict() {'result': [2, 4, 6]} >>> df.select(decorator_double_udf(col("x")).alias("result")).to_pydict() {'result': [3, 6, 9]} """ def _function( func: Callable[..., _R], input_fields: Sequence[pa.DataType | pa.Field] | pa.DataType | pa.Field, return_field: pa.DataType | pa.Field, volatility: Volatility | str, name: str | None = None, ) -> ScalarUDF: if not callable(func): msg = "`func` argument must be callable" raise TypeError(msg) if name is None: if hasattr(func, "__qualname__"): name = func.__qualname__.lower() else: name = func.__class__.__name__.lower() input_fields = data_types_or_fields_to_field_list(input_fields) return_field = data_type_or_field_to_field(return_field, "value") return ScalarUDF( name=name, func=func, input_fields=input_fields, return_field=return_field, volatility=volatility, ) def _decorator( input_fields: Sequence[pa.DataType | pa.Field] | pa.DataType | pa.Field, return_field: pa.DataType | pa.Field, volatility: Volatility | str, name: str | None = None, ) -> Callable: def decorator(func: Callable) -> Callable: udf_caller = ScalarUDF.udf( func, input_fields, return_field, volatility, name ) @functools.wraps(func) def wrapper(*args: Any, **kwargs: Any) -> Callable: return udf_caller(*args, **kwargs) return wrapper return decorator if hasattr(args[0], "__datafusion_scalar_udf__"): return ScalarUDF.from_pycapsule(args[0]) if args and callable(args[0]): # Case 1: Used as a function, require the first parameter to be callable return _function(*args, **kwargs) # Case 2: Used as a decorator with parameters return _decorator(*args, **kwargs) @staticmethod def from_pycapsule(func: ScalarUDFExportable) -> ScalarUDF: """Create a Scalar UDF from ScalarUDF PyCapsule object. This function will instantiate a Scalar UDF that uses a DataFusion ScalarUDF that is exported via the FFI bindings. """ name = str(func.__class__) return ScalarUDF( name=name, func=func, input_fields=None, return_field=None, volatility=None, ) class Accumulator(metaclass=ABCMeta): """Defines how an :py:class:`AggregateUDF` accumulates values.""" @abstractmethod def state(self) -> list[pa.Scalar]: """Return the current state. While this function template expects PyArrow Scalar values return type, you can return any value that can be converted into a Scalar. This includes basic Python data types such as integers and strings. In addition to primitive types, we currently support PyArrow, nanoarrow, and arro3 objects in addition to primitive data types. Other objects that support the Arrow FFI standard will be given a "best attempt" at conversion to scalar objects. """ @abstractmethod def update(self, *values: pa.Array) -> None: """Evaluate an array of values and update state.""" @abstractmethod def merge(self, states: list[pa.Array]) -> None: """Merge a set of states.""" @abstractmethod def evaluate(self) -> pa.Scalar: """Return the resultant value. While this function template expects a PyArrow Scalar value return type, you can return any value that can be converted into a Scalar. This includes basic Python data types such as integers and strings. In addition to primitive types, we currently support PyArrow, nanoarrow, and arro3 objects in addition to primitive data types. Other objects that support the Arrow FFI standard will be given a "best attempt" at conversion to scalar objects. """ class AggregateUDFExportable(Protocol): """Type hint for object that has __datafusion_aggregate_udf__ PyCapsule.""" def __datafusion_aggregate_udf__(self) -> object: ... # noqa: D105 class AggregateUDF: """Class for performing scalar user-defined functions (UDF). Aggregate UDFs operate on a group of rows and return a single value. See also :py:class:`ScalarUDF` for operating on a row by row basis. """ @overload def __init__( self, name: str, accumulator: Callable[[], Accumulator], input_types: list[pa.DataType], return_type: pa.DataType, state_type: list[pa.DataType], volatility: Volatility | str, ) -> None: ... @overload def __init__( self, name: str, accumulator: AggregateUDFExportable, input_types: None = ..., return_type: None = ..., state_type: None = ..., volatility: None = ..., ) -> None: ... def __init__( self, name: str, accumulator: Callable[[], Accumulator] | AggregateUDFExportable, input_types: list[pa.DataType] | None, return_type: pa.DataType | None, state_type: list[pa.DataType] | None, volatility: Volatility | str | None, ) -> None: """Instantiate a user-defined aggregate function (UDAF). See :py:func:`udaf` for a convenience function and argument descriptions. """ if hasattr(accumulator, "__datafusion_aggregate_udf__"): self._udaf = df_internal.AggregateUDF.from_pycapsule(accumulator) return if ( input_types is None or return_type is None or state_type is None or volatility is None ): msg = ( "`input_types`, `return_type`, `state_type`, and `volatility` " "must be provided when `accumulator` is callable." ) raise TypeError(msg) self._udaf = df_internal.AggregateUDF( name, accumulator, input_types, return_type, state_type, str(volatility), ) @classmethod def _from_internal(cls, internal: df_internal.AggregateUDF) -> AggregateUDF: """Wrap an already-constructed internal ``AggregateUDF`` handle. Used by :py:meth:`SessionContext.udaf` to surface a function looked up from the session's function registry without re-running :py:meth:`__init__`. """ wrapper = cls.__new__(cls) wrapper._udaf = internal return wrapper @property def name(self) -> str: """Return the registered name of this UDAF. For UDAFs imported via the FFI capsule protocol, this is the name the capsule itself reports — not the ``name`` argument passed to the constructor (which is ignored on the FFI path). """ return self._udaf.name def __repr__(self) -> str: """Print a string representation of the Aggregate UDF.""" return self._udaf.__repr__() def __call__(self, *args: Expr) -> Expr: """Execute the UDAF. This function is not typically called by an end user. These calls will occur during the evaluation of the dataframe. """ args_raw = [arg.expr for arg in args] return Expr(self._udaf.__call__(*args_raw)) @overload @staticmethod def udaf( input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, state_type: list[pa.DataType], volatility: Volatility | str, name: str | None = None, ) -> Callable[..., AggregateUDF]: ... @overload @staticmethod def udaf( accum: Callable[[], Accumulator], input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, state_type: list[pa.DataType], volatility: Volatility | str, name: str | None = None, ) -> AggregateUDF: ... @overload @staticmethod def udaf(accum: AggregateUDFExportable) -> AggregateUDF: ... @overload @staticmethod def udaf(accum: _PyCapsule) -> AggregateUDF: ... @staticmethod def udaf(*args: Any, **kwargs: Any): # noqa: D417, C901 """Create a new User-Defined Aggregate Function (UDAF). This class allows you to define an aggregate function that can be used in data aggregation or window function calls. Usage: - As a function: ``udaf(accum, input_types, return_type, state_type, volatility, name)``. - As a decorator: ``@udaf(input_types, return_type, state_type, volatility, name)``. When using ``udaf`` as a decorator, do not pass ``accum`` explicitly. If your :py:class:`Accumulator` can be instantiated with no arguments, you can simply pass its type as ``accum``. If you need to pass additional arguments to its constructor, you can define a lambda or a factory method. During runtime the :py:class:`Accumulator` will be constructed for every instance in which this UDAF is used. Examples: >>> import pyarrow.compute as pc >>> from datafusion.user_defined import AggregateUDF, Accumulator, udaf >>> class Summarize(Accumulator): ... def __init__(self, bias: float = 0.0): ... self._sum = pa.scalar(bias) ... def state(self): ... return [self._sum] ... def update(self, values): ... self._sum = pa.scalar( ... self._sum.as_py() + pc.sum(values).as_py()) ... def merge(self, states): ... self._sum = pa.scalar( ... self._sum.as_py() + pc.sum(states[0]).as_py()) ... def evaluate(self): ... return self._sum Using ``udaf`` as a function: >>> udaf1 = AggregateUDF.udaf( ... Summarize, pa.float64(), pa.float64(), ... [pa.float64()], "immutable") Wrapping ``udaf`` with a function: >>> def sum_bias_10() -> Summarize: ... return Summarize(10.0) >>> udaf2 = udaf(sum_bias_10, pa.float64(), pa.float64(), [pa.float64()], ... "immutable") Using ``udaf`` with lambda: >>> udaf3 = udaf(lambda: Summarize(20.0), pa.float64(), pa.float64(), ... [pa.float64()], "immutable") Using ``udaf`` as a decorator: >>> @AggregateUDF.udaf( ... pa.float64(), pa.float64(), ... [pa.float64()], "immutable") ... def udaf4(): ... return Summarize(10.0) Apply to a dataframe: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) >>> df.aggregate([], [udaf1(col("a")).alias("total")]).collect_column( ... "total")[0].as_py() 6.0 >>> df.aggregate([], [udaf2(col("a")).alias("total")]).collect_column( ... "total")[0].as_py() 16.0 >>> df.aggregate([], [udaf3(col("a")).alias("total")]).collect_column( ... "total")[0].as_py() 26.0 >>> df.aggregate([], [udaf4(col("a")).alias("total")]).collect_column( ... "total")[0].as_py() 16.0 Args: accum: The accumulator python function. Only needed when calling as a function. Skip this argument when using ``udaf`` as a decorator. If you have a Rust backed AggregateUDF within a PyCapsule, you can pass this parameter and ignore the rest. They will be determined directly from the underlying function. See the online documentation for more information. input_types: The data types of the arguments to ``accum``. return_type: The data type of the return value. state_type: The data types of the intermediate accumulation. volatility: See :py:class:`Volatility` for allowed values. name: A descriptive name for the function. Returns: A user-defined aggregate function, which can be used in either data aggregation or window function calls. """ # noqa: E501 W505 def _function( accum: Callable[[], Accumulator], input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, state_type: list[pa.DataType], volatility: Volatility | str, name: str | None = None, ) -> AggregateUDF: if not callable(accum): msg = "`func` must be callable." raise TypeError(msg) if not isinstance(accum(), Accumulator): msg = "Accumulator must implement the abstract base class Accumulator" raise TypeError(msg) if name is None: name = accum().__class__.__qualname__.lower() if isinstance(input_types, pa.DataType): input_types = [input_types] return AggregateUDF( name=name, accumulator=accum, input_types=input_types, return_type=return_type, state_type=state_type, volatility=volatility, ) def _decorator( input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, state_type: list[pa.DataType], volatility: Volatility | str, name: str | None = None, ) -> Callable[..., Callable[..., Expr]]: def decorator(accum: Callable[[], Accumulator]) -> Callable[..., Expr]: udaf_caller = AggregateUDF.udaf( accum, input_types, return_type, state_type, volatility, name ) @functools.wraps(accum) def wrapper(*args: Any, **kwargs: Any) -> Expr: return udaf_caller(*args, **kwargs) return wrapper return decorator if hasattr(args[0], "__datafusion_aggregate_udf__") or _is_pycapsule(args[0]): return AggregateUDF.from_pycapsule(args[0]) if args and callable(args[0]): # Case 1: Used as a function, require the first parameter to be callable return _function(*args, **kwargs) # Case 2: Used as a decorator with parameters return _decorator(*args, **kwargs) @staticmethod def from_pycapsule(func: AggregateUDFExportable | _PyCapsule) -> AggregateUDF: """Create an Aggregate UDF from AggregateUDF PyCapsule object. This function will instantiate a Aggregate UDF that uses a DataFusion AggregateUDF that is exported via the FFI bindings. """ if _is_pycapsule(func): aggregate = cast("AggregateUDF", object.__new__(AggregateUDF)) aggregate._udaf = df_internal.AggregateUDF.from_pycapsule(func) return aggregate capsule = cast("AggregateUDFExportable", func) name = str(capsule.__class__) return AggregateUDF( name=name, accumulator=capsule, input_types=None, return_type=None, state_type=None, volatility=None, ) class WindowEvaluator: """Evaluator class for user-defined window functions (UDWF). It is up to the user to decide which evaluate function is appropriate. +------------------------+--------------------------------+------------------+---------------------------+ | ``uses_window_frame`` | ``supports_bounded_execution`` | ``include_rank`` | function_to_implement | +========================+================================+==================+===========================+ | False (default) | False (default) | False (default) | ``evaluate_all`` | +------------------------+--------------------------------+------------------+---------------------------+ | False | True | False | ``evaluate`` | +------------------------+--------------------------------+------------------+---------------------------+ | False | True/False | True | ``evaluate_all_with_rank``| +------------------------+--------------------------------+------------------+---------------------------+ | True | True/False | True/False | ``evaluate`` | +------------------------+--------------------------------+------------------+---------------------------+ """ # noqa: W505, E501 def memoize(self) -> None: """Perform a memoize operation to improve performance. When the window frame has a fixed beginning (e.g UNBOUNDED PRECEDING), some functions such as FIRST_VALUE and NTH_VALUE do not need the (unbounded) input once they have seen a certain amount of input. `memoize` is called after each input batch is processed, and such functions can save whatever they need """ def get_range(self, idx: int, num_rows: int) -> tuple[int, int]: # noqa: ARG002 """Return the range for the window function. If `uses_window_frame` flag is `false`. This method is used to calculate required range for the window function during stateful execution. Generally there is no required range, hence by default this returns smallest range(current row). e.g seeing current row is enough to calculate window result (such as row_number, rank, etc) Args: idx:: Current index num_rows: Number of rows. """ return (idx, idx + 1) def is_causal(self) -> bool: """Get whether evaluator needs future data for its result.""" return False def evaluate_all(self, values: list[pa.Array], num_rows: int) -> pa.Array: """Evaluate a window function on an entire input partition. This function is called once per input *partition* for window functions that *do not use* values from the window frame, such as :py:func:`~datafusion.functions.row_number`, :py:func:`~datafusion.functions.rank`, :py:func:`~datafusion.functions.dense_rank`, :py:func:`~datafusion.functions.percent_rank`, :py:func:`~datafusion.functions.cume_dist`, :py:func:`~datafusion.functions.lead`, and :py:func:`~datafusion.functions.lag`. It produces the result of all rows in a single pass. It expects to receive the entire partition as the ``value`` and must produce an output column with one output row for every input row. ``num_rows`` is required to correctly compute the output in case ``len(values) == 0`` Implementing this function is an optimization. Certain window functions are not affected by the window frame definition or the query doesn't have a frame, and ``evaluate`` skips the (costly) window frame boundary calculation and the overhead of calling ``evaluate`` for each output row. For example, the `LAG` built in window function does not use the values of its window frame (it can be computed in one shot on the entire partition with ``Self::evaluate_all`` regardless of the window defined in the ``OVER`` clause) .. code-block:: text lag(x, 1) OVER (ORDER BY z ROWS BETWEEN 2 PRECEDING AND 3 FOLLOWING) However, ``avg()`` computes the average in the window and thus does use its window frame. .. code-block:: text avg(x) OVER (PARTITION BY y ORDER BY z ROWS BETWEEN 2 PRECEDING AND 3 FOLLOWING) """ # noqa: W505, E501 def evaluate( self, values: list[pa.Array], eval_range: tuple[int, int] ) -> pa.Scalar: """Evaluate window function on a range of rows in an input partition. This is the simplest and most general function to implement but also the least performant as it creates output one row at a time. It is typically much faster to implement stateful evaluation using one of the other specialized methods on this trait. Returns a [`ScalarValue`] that is the value of the window function within `range` for the entire partition. Argument `values` contains the evaluation result of function arguments and evaluation results of ORDER BY expressions. If function has a single argument, `values[1..]` will contain ORDER BY expression results. """ def evaluate_all_with_rank( self, num_rows: int, ranks_in_partition: list[tuple[int, int]] ) -> pa.Array: """Called for window functions that only need the rank of a row. Evaluate the partition evaluator against the partition using the row ranks. For example, ``rank(col("a"))`` produces .. code-block:: text a | rank - + ---- A | 1 A | 1 C | 3 D | 4 D | 4 For this case, `num_rows` would be `5` and the `ranks_in_partition` would be called with .. code-block:: text [ (0,1), (2,2), (3,4), ] The user must implement this method if ``include_rank`` returns True. """ def supports_bounded_execution(self) -> bool: """Can the window function be incrementally computed using bounded memory?""" return False def uses_window_frame(self) -> bool: """Does the window function use the values from the window frame?""" return False def include_rank(self) -> bool: """Can this function be evaluated with (only) rank?""" return False class WindowUDFExportable(Protocol): """Type hint for object that has __datafusion_window_udf__ PyCapsule.""" def __datafusion_window_udf__(self) -> object: ... # noqa: D105 class WindowUDF: """Class for performing window user-defined functions (UDF). Window UDFs operate on a partition of rows. See also :py:class:`ScalarUDF` for operating on a row by row basis. """ def __init__( self, name: str, func: Callable[[], WindowEvaluator], input_types: list[pa.DataType], return_type: pa.DataType, volatility: Volatility | str, ) -> None: """Instantiate a user-defined window function (UDWF). See :py:func:`udwf` for a convenience function and argument descriptions. """ if hasattr(func, "__datafusion_window_udf__"): self._udwf = df_internal.WindowUDF.from_pycapsule(func) return self._udwf = df_internal.WindowUDF( name, func, input_types, return_type, str(volatility) ) @classmethod def _from_internal(cls, internal: df_internal.WindowUDF) -> WindowUDF: """Wrap an already-constructed internal ``WindowUDF`` handle. Used by :py:meth:`SessionContext.udwf` to surface a function looked up from the session's function registry without re-running :py:meth:`__init__`. """ wrapper = cls.__new__(cls) wrapper._udwf = internal return wrapper @property def name(self) -> str: """Return the registered name of this UDWF. For UDWFs imported via the FFI capsule protocol, this is the name the capsule itself reports — not the ``name`` argument passed to the constructor (which is ignored on the FFI path). """ return self._udwf.name def __repr__(self) -> str: """Print a string representation of the Window UDF.""" return self._udwf.__repr__() def __call__(self, *args: Expr) -> Expr: """Execute the UDWF. This function is not typically called by an end user. These calls will occur during the evaluation of the dataframe. """ args_raw = [arg.expr for arg in args] return Expr(self._udwf.__call__(*args_raw)) @overload @staticmethod def udwf( input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, volatility: Volatility | str, name: str | None = None, ) -> Callable[..., WindowUDF]: ... @overload @staticmethod def udwf( func: Callable[[], WindowEvaluator], input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, volatility: Volatility | str, name: str | None = None, ) -> WindowUDF: ... @staticmethod def udwf(*args: Any, **kwargs: Any): # noqa: D417 """Create a new User-Defined Window Function (UDWF). This class can be used both as either a function or a decorator. Usage: - As a function: ``udwf(func, input_types, return_type, volatility, name)``. - As a decorator: ``@udwf(input_types, return_type, volatility, name)``. When using ``udwf`` as a decorator, do not pass ``func`` explicitly. Examples: >>> from datafusion.user_defined import WindowUDF, WindowEvaluator, udwf >>> class BiasedNumbers(WindowEvaluator): ... def __init__(self, start: int = 0): ... self.start = start ... def evaluate_all(self, values, num_rows): ... return pa.array( ... [self.start + i for i in range(num_rows)]) Using ``udwf`` as a function: >>> udwf1 = WindowUDF.udwf( ... BiasedNumbers, pa.int64(), pa.int64(), "immutable") >>> def bias_10() -> BiasedNumbers: ... return BiasedNumbers(10) >>> udwf2 = udwf(bias_10, pa.int64(), pa.int64(), "immutable") >>> udwf3 = udwf( ... lambda: BiasedNumbers(20), pa.int64(), pa.int64(), "immutable" ... ) Using ``udwf`` as a decorator: >>> @WindowUDF.udwf(pa.int64(), pa.int64(), "immutable") ... def biased_numbers(): ... return BiasedNumbers(10) Apply to a dataframe: >>> ctx = dfn.SessionContext() >>> df = ctx.from_pydict({"a": [10, 20, 30]}) >>> df.select(udwf1(col("a")).alias("result")).to_pydict() {'result': [0, 1, 2]} >>> df.select(udwf2(col("a")).alias("result")).to_pydict() {'result': [10, 11, 12]} >>> df.select(udwf3(col("a")).alias("result")).to_pydict() {'result': [20, 21, 22]} >>> df.select(biased_numbers(col("a")).alias("result")).to_pydict() {'result': [10, 11, 12]} Args: func: Only needed when calling as a function. Skip this argument when using ``udwf`` as a decorator. If you have a Rust backed WindowUDF within a PyCapsule, you can pass this parameter and ignore the rest. They will be determined directly from the underlying function. See the online documentation for more information. input_types: The data types of the arguments. return_type: The data type of the return value. volatility: See :py:class:`Volatility` for allowed values. name: A descriptive name for the function. Returns: A user-defined window function that can be used in window function calls. """ if hasattr(args[0], "__datafusion_window_udf__"): return WindowUDF.from_pycapsule(args[0]) if args and callable(args[0]): # Case 1: Used as a function, require the first parameter to be callable return WindowUDF._create_window_udf(*args, **kwargs) # Case 2: Used as a decorator with parameters return WindowUDF._create_window_udf_decorator(*args, **kwargs) @staticmethod def _create_window_udf( func: Callable[[], WindowEvaluator], input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, volatility: Volatility | str, name: str | None = None, ) -> WindowUDF: """Create a WindowUDF instance from function arguments.""" if not callable(func): msg = "`func` must be callable." raise TypeError(msg) if not isinstance(func(), WindowEvaluator): msg = "`func` must implement the abstract base class WindowEvaluator" raise TypeError(msg) name = name or func.__qualname__.lower() input_types = ( [input_types] if isinstance(input_types, pa.DataType) else input_types ) return WindowUDF(name, func, input_types, return_type, volatility) @staticmethod def _get_default_name(func: Callable) -> str: """Get the default name for a function based on its attributes.""" if hasattr(func, "__qualname__"): return func.__qualname__.lower() return func.__class__.__name__.lower() @staticmethod def _normalize_input_types( input_types: pa.DataType | list[pa.DataType], ) -> list[pa.DataType]: """Convert a single DataType to a list if needed.""" if isinstance(input_types, pa.DataType): return [input_types] return input_types @staticmethod def _create_window_udf_decorator( input_types: pa.DataType | list[pa.DataType], return_type: pa.DataType, volatility: Volatility | str, name: str | None = None, ) -> Callable[[Callable[[], WindowEvaluator]], Callable[..., Expr]]: """Create a decorator for a WindowUDF.""" def decorator(func: Callable[[], WindowEvaluator]) -> Callable[..., Expr]: udwf_caller = WindowUDF._create_window_udf( func, input_types, return_type, volatility, name ) @functools.wraps(func) def wrapper(*args: Any, **kwargs: Any) -> Expr: return udwf_caller(*args, **kwargs) return wrapper return decorator @staticmethod def from_pycapsule(func: WindowUDFExportable) -> WindowUDF: """Create a Window UDF from WindowUDF PyCapsule object. This function will instantiate a Window UDF that uses a DataFusion WindowUDF that is exported via the FFI bindings. """ name = str(func.__class__) return WindowUDF( name=name, func=func, input_types=None, return_type=None, volatility=None, ) def _wrap_session_kwarg_for_udtf(func: Callable[..., Any]) -> Callable[..., Any]: """Adapt the raw internal session pyo3 object back to a Python wrapper. The Rust call site forwards a ``datafusion._internal.SessionContext``, but UDTF authors expect to interact with the public :class:`datafusion.SessionContext` wrapper. This closure wraps the internal object once per call before delegating to ``func``. """ @functools.wraps(func, updated=()) def adapter(*args: Any, session: Any, **kwargs: Any) -> Any: wrapped = SessionContext.__new__(SessionContext) wrapped.ctx = session return func(*args, session=wrapped, **kwargs) return adapter class TableFunction: """Class for performing user-defined table functions (UDTF). Table functions generate new table providers based on the input expressions. """ def __init__( self, name: str, func: Callable[..., Any], ctx: SessionContext | None = None, *, with_session: bool = False, ) -> None: """Instantiate a user-defined table function (UDTF). Set ``with_session=True`` to have the calling :class:`SessionContext` passed as a ``session`` keyword argument on each invocation. Use it inside the callback to look up registered tables, UDFs, or session configuration. When ``with_session`` is ``False`` (the default), ``func`` is invoked with the positional expression arguments only. ``with_session=True`` is only supported for pure-Python callables. Passing it together with an FFI-exported table function (one exposing ``__datafusion_table_function__``) raises :class:`TypeError`. Registry mutations performed through the injected session (such as registering tables or UDFs) propagate to the caller's :class:`SessionContext` because the registries are shared. Configuration changes do **not** propagate; the wrapper holds its own clone of the session config. See :py:func:`udtf` for a convenience function and argument descriptions. """ if with_session and hasattr(func, "__datafusion_table_function__"): msg = ( "`with_session=True` is not supported for FFI-exported table " "functions; session injection requires a pure-Python callable." ) raise TypeError(msg) registered = _wrap_session_kwarg_for_udtf(func) if with_session else func self._udtf = df_internal.TableFunction(name, registered, ctx, with_session) def __call__(self, *args: Expr) -> Any: """Execute the UDTF and return a table provider.""" args_raw = [arg.expr for arg in args] return self._udtf.__call__(*args_raw) @overload @staticmethod def udtf( name: str, *, with_session: bool = False, ) -> Callable[..., Any]: ... @overload @staticmethod def udtf( func: Callable[..., Any], name: str, *, with_session: bool = False, ) -> TableFunction: ... @staticmethod def udtf(*args: Any, with_session: bool = False, **kwargs: Any): """Create a new User-Defined Table Function (UDTF). Pass ``with_session=True`` to have the calling :class:`SessionContext` injected as a ``session`` keyword argument on each invocation. """ if args and callable(args[0]): # Case 1: Used as a function, require the first parameter to be callable return TableFunction._create_table_udf( *args, with_session=with_session, **kwargs ) if args and hasattr(args[0], "__datafusion_table_function__"): # Case 2: We have a datafusion FFI provided function if with_session: msg = ( "`with_session=True` is not supported for FFI-exported " "table functions; session injection requires a " "pure-Python callable." ) raise TypeError(msg) return TableFunction(args[1], args[0]) # Case 3: Used as a decorator with parameters return TableFunction._create_table_udf_decorator( *args, with_session=with_session, **kwargs ) @staticmethod def _create_table_udf( func: Callable[..., Any], name: str, *, with_session: bool = False, ) -> TableFunction: """Create a TableFunction instance from function arguments.""" if not callable(func): msg = "`func` must be callable." raise TypeError(msg) return TableFunction(name, func, with_session=with_session) @staticmethod def _create_table_udf_decorator( name: str | None = None, *, with_session: bool = False, ) -> Callable[[Callable[..., Any]], TableFunction]: """Create a decorator for a TableFunction.""" def decorator(func: Callable[..., Any]) -> TableFunction: return TableFunction._create_table_udf( func, name, with_session=with_session ) return decorator def __repr__(self) -> str: """User printable representation.""" return self._udtf.__repr__() # Convenience exports so we can import instead of treating as # variables at the package root udf = ScalarUDF.udf udaf = AggregateUDF.udaf udwf = WindowUDF.udwf udtf = TableFunction.udtf apache-datafusion-python-54.0.0/python/tests/000077500000000000000000000000001521727353300211665ustar00rootroot00000000000000apache-datafusion-python-54.0.0/python/tests/__init__.py000066400000000000000000000014211521727353300232750ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. apache-datafusion-python-54.0.0/python/tests/_pickle_multiprocessing_helpers.py000066400000000000000000000055671521727353300302140ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. # The leading underscore is load-bearing: pytest with --import-mode=importlib # (used in CI) assigns synthetic module names to test modules, which breaks # subprocess imports during multiprocessing. An underscore-prefixed module is # not collected as a test module, so it imports under its normal __name__ # inside worker processes. from __future__ import annotations import pyarrow as pa from datafusion import SessionContext, udf from datafusion.ipc import clear_worker_ctx, set_worker_ctx def make_double_udf(): """Build the canonical UDF used in the multiprocessing tests.""" return udf( lambda arr: pa.array([(v.as_py() or 0) * 2 for v in arr]), [pa.int64()], pa.int64(), volatility="immutable", name="double", ) def make_times_seven_udf(): """Closure-capturing UDF — verifies cloudpickle preserves closed-over state.""" multiplier = 7 def fn(arr): return pa.array([(v.as_py() or 0) * multiplier for v in arr]) return udf( fn, [pa.int64()], pa.int64(), volatility="immutable", name="times_seven", ) def init_worker_empty(): """Pool initializer: install an empty SessionContext (no UDFs).""" set_worker_ctx(SessionContext()) def init_worker_clear(): """Pool initializer: explicitly clear any prior worker context.""" clear_worker_ctx() def unpickle_and_describe(blob: bytes) -> str: """Unpickle a proto-bytes blob and return its canonical name.""" import pickle expr = pickle.loads(blob) # noqa: S301 return expr.canonical_name() def unpickle_and_evaluate(blob: bytes, batch: list[int]) -> list[int]: """Unpickle an expression and evaluate it against an in-memory batch. Returns the result column as a Python list. Used to verify that cloudpickled UDFs (including closure state) execute correctly in a fresh worker process. """ import pickle expr = pickle.loads(blob) # noqa: S301 ctx = SessionContext() df = ctx.from_pydict({"a": batch}) out = df.with_column("result", expr).select("result") return out.to_pydict()["result"] apache-datafusion-python-54.0.0/python/tests/conftest.py000066400000000000000000000033301521727353300233640ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import pyarrow as pa import pytest from datafusion import DataFrame, SessionContext from pyarrow.csv import write_csv @pytest.fixture def ctx(): return SessionContext() @pytest.fixture def database(ctx, tmp_path): path = tmp_path / "test.csv" table = pa.Table.from_arrays( [ [1, 2, 3, 4], ["a", "b", "c", "d"], [1.1, 2.2, 3.3, 4.4], ], names=["int", "str", "float"], ) write_csv(table, path) ctx.register_csv("csv", path) ctx.register_csv("csv1", str(path)) ctx.register_csv( "csv2", path, has_header=True, delimiter=",", schema_infer_max_records=10, ) @pytest.fixture def fail_collect(monkeypatch): def _fail_collect(self, *args, **kwargs): # pragma: no cover - failure path msg = "collect should not be called" raise AssertionError(msg) monkeypatch.setattr(DataFrame, "collect", _fail_collect) apache-datafusion-python-54.0.0/python/tests/data_test_context/000077500000000000000000000000001521727353300247025ustar00rootroot00000000000000apache-datafusion-python-54.0.0/python/tests/data_test_context/data.json000066400000000000000000000000711521727353300265040ustar00rootroot00000000000000{"A": "a", "B": 1} {"A": "b", "B": 2} {"A": "c", "B": 3} apache-datafusion-python-54.0.0/python/tests/generic.py000066400000000000000000000044611521727353300231610ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import datetime from datetime import timezone import numpy as np import pyarrow as pa import pyarrow.csv # used to write parquet files import pyarrow.parquet as pq def data(): rng = np.random.default_rng(1) data = np.concatenate( [ rng.normal(0, 0.01, size=50), rng.normal(50, 0.01, size=50), ] ) return pa.array(data) def data_with_nans(): rng = np.random.default_rng(0) data = rng.normal(0, 0.01, size=50) mask = rng.normal(0, 2, size=50) data[mask == 0] = np.nan return data def data_datetime(f): data = [ datetime.datetime.now(tz=timezone.utc), datetime.datetime.now(tz=timezone.utc) - datetime.timedelta(days=1), datetime.datetime.now(tz=timezone.utc) + datetime.timedelta(days=1), ] return pa.array(data, type=pa.timestamp(f), mask=np.array([False, True, False])) def data_date32(): data = [ datetime.date(2000, 1, 1), datetime.date(1980, 1, 1), datetime.date(2030, 1, 1), ] return pa.array(data, type=pa.date32(), mask=np.array([False, True, False])) def data_timedelta(f): data = [ datetime.timedelta(days=100), datetime.timedelta(days=1), datetime.timedelta(seconds=1), ] return pa.array(data, type=pa.duration(f), mask=np.array([False, True, False])) def data_binary_other(): return np.array([1, 0, 0], dtype="u4") def write_parquet(path, data): table = pa.Table.from_arrays([data], names=["a"]) pq.write_table(table, path) return str(path) apache-datafusion-python-54.0.0/python/tests/test_aggregation.py000066400000000000000000000401771521727353300250770ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import numpy as np import pyarrow as pa import pytest from datafusion import SessionContext, column, lit from datafusion import functions as f from datafusion.common import NullTreatment @pytest.fixture def df(): ctx = SessionContext() # create a RecordBatch and a new DataFrame from it batch = pa.RecordBatch.from_arrays( [ pa.array([1, 2, 3]), pa.array([4, 4, 6]), pa.array([9, 8, 5]), pa.array([True, True, False]), pa.array([1, 2, None]), ], names=["a", "b", "c", "d", "e"], ) return ctx.create_dataframe([[batch]]) @pytest.fixture def df_partitioned(): ctx = SessionContext() # create a RecordBatch and a new DataFrame from it batch = pa.RecordBatch.from_arrays( [ pa.array([0, 1, 2, 3, 4, 5, 6]), pa.array([7, None, 7, 8, 9, None, 9]), pa.array(["A", "A", "A", "A", "B", "B", "B"]), ], names=["a", "b", "c"], ) return ctx.create_dataframe([[batch]]) @pytest.fixture def df_aggregate_100(): ctx = SessionContext() ctx.register_csv("aggregate_test_data", "./testing/data/csv/aggregate_test_100.csv") return ctx.table("aggregate_test_data") @pytest.mark.parametrize( ("agg_expr", "calc_expected"), [ (f.avg(column("a")), lambda a, b, c, d: np.array(np.average(a))), ( f.corr(column("a"), column("b")), lambda a, b, c, d: np.array(np.corrcoef(a, b)[0][1]), ), (f.count(column("a")), lambda a, b, c, d: pa.array([len(a)])), # Sample (co)variance -> ddof=1 # Population (co)variance -> ddof=0 ( f.covar(column("a"), column("b")), lambda a, b, c, d: np.array(np.cov(a, b, ddof=1)[0][1]), ), ( f.covar_pop(column("a"), column("c")), lambda a, b, c, d: np.array(np.cov(a, c, ddof=0)[0][1]), ), ( f.covar_samp(column("b"), column("c")), lambda a, b, c, d: np.array(np.cov(b, c, ddof=1)[0][1]), ), # f.grouping(col_a), # noqa: ERA001 No physical plan implemented yet (f.max(column("a")), lambda a, b, c, d: np.array(np.max(a))), (f.mean(column("b")), lambda a, b, c, d: np.array(np.mean(b))), (f.median(column("b")), lambda a, b, c, d: np.array(np.median(b))), (f.min(column("a")), lambda a, b, c, d: np.array(np.min(a))), (f.sum(column("b")), lambda a, b, c, d: np.array(np.sum(b.to_pylist()))), # Sample stdev -> ddof=1 # Population stdev -> ddof=0 (f.stddev(column("a")), lambda a, b, c, d: np.array(np.std(a, ddof=1))), (f.stddev_pop(column("b")), lambda a, b, c, d: np.array(np.std(b, ddof=0))), (f.stddev_samp(column("c")), lambda a, b, c, d: np.array(np.std(c, ddof=1))), (f.var(column("a")), lambda a, b, c, d: np.array(np.var(a, ddof=1))), (f.var_pop(column("b")), lambda a, b, c, d: np.array(np.var(b, ddof=0))), (f.var_samp(column("c")), lambda a, b, c, d: np.array(np.var(c, ddof=1))), ], ) def test_aggregation_stats(df, agg_expr, calc_expected): df = df.select("a", "b", "c", "d") agg_df = df.aggregate([], [agg_expr]) result = agg_df.collect()[0] values_a, values_b, values_c, values_d = df.collect()[0] expected = calc_expected(values_a, values_b, values_c, values_d) np.testing.assert_array_almost_equal(result.column(0), expected) @pytest.mark.parametrize( ("agg_expr", "expected", "array_sort"), [ (f.approx_distinct(column("b")), pa.array([2], type=pa.uint64()), False), ( f.approx_distinct( column("b"), filter=column("a") != lit(3), ), pa.array([1], type=pa.uint64()), False, ), (f.approx_median(column("b")), pa.array([4], type=pa.float64()), False), ( f.median(column("b"), distinct=True), pa.array([5], type=pa.float64()), False, ), ( f.median(column("b"), filter=column("a") != 2), pa.array([5], type=pa.float64()), False, ), ( f.approx_median(column("b"), filter=column("a") != 2), pa.array([5], type=pa.float64()), False, ), ( f.approx_percentile_cont(column("b"), 0.5), pa.array([4], type=pa.float64()), False, ), ( f.approx_percentile_cont( column("b").sort(ascending=True, nulls_first=False), 0.5, num_centroids=2, ), pa.array([4.75], type=pa.float64()), False, ), ( f.approx_percentile_cont_with_weight(column("b"), lit(0.6), 0.5), pa.array([4], type=pa.float64()), False, ), ( f.approx_percentile_cont_with_weight( column("b").sort(ascending=False, nulls_first=False), lit(0.6), 0.5 ), pa.array([4], type=pa.float64()), False, ), ( f.approx_percentile_cont_with_weight( column("b"), lit(0.6), 0.5, filter=column("a") != lit(3) ), pa.array([4], type=pa.float64()), False, ), (f.array_agg(column("b")), pa.array([[4, 4, 6]]), False), (f.array_agg(column("b"), distinct=True), pa.array([[4, 6]]), True), ( f.array_agg(column("e"), filter=column("e").is_not_null()), pa.array([[1, 2]]), False, ), ( f.array_agg(column("b"), order_by=[column("c")]), pa.array([[6, 4, 4]]), False, ), ( f.array_agg(column("b"), order_by=column("c")), pa.array([[6, 4, 4]]), False, ), (f.avg(column("b"), filter=column("a") != lit(1)), pa.array([5.0]), False), (f.avg(column("b"), distinct=True), pa.array([5.0]), False), (f.sum(column("b"), filter=column("a") != lit(1)), pa.array([10]), False), (f.sum(column("b"), distinct=True), pa.array([10]), False), (f.count(column("b"), distinct=True), pa.array([2]), False), (f.count(column("b"), filter=column("a") != 3), pa.array([2]), False), (f.count(), pa.array([3]), False), (f.count(column("e")), pa.array([2]), False), (f.count_star(filter=column("a") != 3), pa.array([2]), False), (f.max(column("a"), filter=column("a") != lit(3)), pa.array([2]), False), (f.min(column("a"), filter=column("a") != lit(1)), pa.array([2]), False), ( f.stddev(column("a"), filter=column("a") != lit(2)), pa.array([np.sqrt(2)]), False, ), ( f.stddev_pop(column("a"), filter=column("a") != lit(2)), pa.array([1.0]), False, ), ], ) def test_aggregation(df, agg_expr, expected, array_sort): agg_df = df.aggregate([], [agg_expr.alias("agg_expr")]) if array_sort: agg_df = agg_df.select(f.array_sort(column("agg_expr"))) agg_df.show() result = agg_df.collect()[0] assert result.column(0) == expected @pytest.mark.parametrize( ("name", "expr", "expected"), [ ( "approx_percentile_cont", f.approx_percentile_cont(column("c3"), 0.95, num_centroids=200), [73.55, 68.0, 122.5, 124.2, 115.6], ), ( "approx_perc_cont_few_centroids", f.approx_percentile_cont(column("c3"), 0.95, num_centroids=5), [72.775, 68.0, 119.4075, 124.825, 115.44], ), ( "approx_perc_cont_filtered", f.approx_percentile_cont( column("c3"), 0.95, num_centroids=200, filter=column("c3") > lit(0) ), [83.0, 68.0, 122.75, 124.9, 117.6], ), ( "corr", f.corr(column("c3"), column("c2")), [-0.1056, -0.2808, 0.0023, 0.0022, -0.2473], ), ( "corr_w_filter", f.corr(column("c3"), column("c2"), filter=column("c3") > lit(0)), [-0.3298, 0.2925, 0.2467, -0.2269, 0.0358], ), ( "covar_pop", f.covar_pop(column("c3"), column("c2")), [-7.2857, -25.6731, 0.2222, 0.2469, -20.2857], ), ( "covar_pop_w_filter", f.covar_pop(column("c3"), column("c2"), filter=column("c3") > lit(0)), [-9.25, 9.0579, 13.7521, -9.9669, 1.1641], ), ( "covar_samp", f.covar_samp(column("c3"), column("c2")), [-7.65, -27.0994, 0.2333, 0.2614, -21.3], ), ( "covar_samp_w_filter", f.covar_samp(column("c3"), column("c2"), filter=column("c3") > lit(0)), [-10.5714, 9.9636, 15.1273, -10.9636, 1.2417], ), ( "var_samp", f.var_samp(column("c2")), [1.9286, 2.2047, 1.6333, 2.1438, 1.6], ), ( "var_samp_w_filter", f.var_samp(column("c2"), filter=column("c3") > lit(0)), [1.4286, 2.4182, 1.8545, 1.4727, 1.6292], ), ( "var_pop", f.var_pop(column("c2")), [1.8367, 2.0886, 1.5556, 2.0247, 1.5238], ), ( "var_pop_w_filter", f.var_pop(column("c2"), filter=column("c3") > lit(0)), [1.25, 2.1983, 1.686, 1.3388, 1.5273], ), ], ) def test_aggregate_100(df_aggregate_100, name, expr, expected): # https://github.com/apache/datafusion/blob/bddb6415a50746d2803dd908d19c3758952d74f9/datafusion/sqllogictest/test_files/aggregate.slt#L1490-L1498 df = ( df_aggregate_100.aggregate( [column("c1")], [expr.alias(name)], ) .select("c1", f.round(column(name), lit(4)).alias(name)) .sort(column("c1").sort(ascending=True)) ) df.show() expected_dict = { "c1": ["a", "b", "c", "d", "e"], name: expected, } assert df.collect()[0].to_pydict() == expected_dict data_test_bitwise_and_boolean_functions = [ ("bit_and", f.bit_and(column("a")), [0]), ("bit_and_filter", f.bit_and(column("a"), filter=column("a") != lit(2)), [1]), ("bit_or", f.bit_or(column("b")), [6]), ("bit_or_filter", f.bit_or(column("b"), filter=column("a") != lit(3)), [4]), ("bit_xor", f.bit_xor(column("c")), [4]), ("bit_xor_distinct", f.bit_xor(column("b"), distinct=True), [2]), ("bit_xor_filter", f.bit_xor(column("b"), filter=column("a") != lit(3)), [0]), ( "bit_xor_filter_distinct", f.bit_xor(column("b"), distinct=True, filter=column("a") != lit(3)), [4], ), ("bool_and", f.bool_and(column("d")), [False]), ("bool_and_filter", f.bool_and(column("d"), filter=column("a") != lit(3)), [True]), ("bool_or", f.bool_or(column("d")), [True]), ("bool_or_filter", f.bool_or(column("d"), filter=column("a") == lit(3)), [False]), ] @pytest.mark.parametrize( ("name", "expr", "result"), data_test_bitwise_and_boolean_functions ) def test_bit_and_bool_fns(df, name, expr, result): df = df.aggregate([], [expr.alias(name)]) expected = { name: result, } assert df.collect()[0].to_pydict() == expected @pytest.mark.parametrize( ("name", "expr", "result"), [ ("first_value", f.first_value(column("a")), [0, 4]), ( "first_value_ordered", f.first_value(column("a"), order_by=[column("a").sort(ascending=False)]), [3, 6], ), ( "first_value_with_null", f.first_value( column("b"), order_by=[column("b").sort(ascending=True)], null_treatment=NullTreatment.RESPECT_NULLS, ), [None, None], ), ( "first_value_no_list_order_by", f.first_value( column("b"), order_by=column("b"), null_treatment=NullTreatment.RESPECT_NULLS, ), [None, None], ), ( "first_value_ignore_null", f.first_value( column("b"), order_by=[column("b").sort(ascending=True)], null_treatment=NullTreatment.IGNORE_NULLS, ), [7, 9], ), ( "last_value_ordered", f.last_value(column("a"), order_by=[column("a").sort(ascending=False)]), [0, 4], ), ( "last_value_no_list_ordered", f.last_value(column("a"), order_by=column("a")), [3, 6], ), ( "last_value_with_null", f.last_value( column("b"), order_by=[column("b").sort(ascending=True, nulls_first=False)], null_treatment=NullTreatment.RESPECT_NULLS, ), [None, None], ), ( "last_value_ignore_null", f.last_value( column("b"), order_by=[column("b").sort(ascending=True)], null_treatment=NullTreatment.IGNORE_NULLS, ), [8, 9], ), ( "nth_value_ordered", f.nth_value(column("a"), 2, order_by=[column("a").sort(ascending=False)]), [2, 5], ), ( "nth_value_no_list_ordered", f.nth_value(column("a"), 2, order_by=column("a").sort(ascending=False)), [2, 5], ), ( "nth_value_with_null", f.nth_value( column("b"), 3, order_by=[column("b").sort(ascending=True, nulls_first=False)], null_treatment=NullTreatment.RESPECT_NULLS, ), [8, None], ), ( "nth_value_ignore_null", f.nth_value( column("b"), 2, order_by=[column("b").sort(ascending=True)], null_treatment=NullTreatment.IGNORE_NULLS, ), [7, 9], ), ], ) def test_first_last_value(df_partitioned, name, expr, result) -> None: df = df_partitioned.aggregate([column("c")], [expr.alias(name)]).sort(column("c")) expected = { "c": ["A", "B"], name: result, } assert df.collect()[0].to_pydict() == expected @pytest.mark.parametrize( ("name", "expr", "result"), [ ("string_agg", f.string_agg(column("a"), ","), "one,two,three,two"), ("string_agg", f.string_agg(column("b"), ""), "03124"), ( "string_agg", f.string_agg(column("a"), ",", filter=column("b") != lit(3)), "one,three,two", ), ( "string_agg", f.string_agg(column("a"), ",", order_by=[column("b")]), "one,three,two,two", ), ( "string_agg", f.string_agg(column("a"), ",", order_by=column("b")), "one,three,two,two", ), ], ) def test_string_agg(name, expr, result) -> None: ctx = SessionContext() df = ctx.from_pydict( { "a": ["one", "two", None, "three", "two"], "b": [0, 3, 1, 2, 4], } ) df = df.aggregate([], [expr.alias(name)]) expected = { name: [result], } df.show() assert df.collect()[0].to_pydict() == expected apache-datafusion-python-54.0.0/python/tests/test_catalog.py000066400000000000000000000254221521727353300242160ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. from __future__ import annotations from typing import TYPE_CHECKING import datafusion as dfn import pyarrow as pa import pyarrow.dataset as ds import pytest from datafusion import Catalog, SessionContext, Table, udtf if TYPE_CHECKING: from datafusion.catalog import CatalogProvider, CatalogProviderExportable # Note we take in `database` as a variable even though we don't use # it because that will cause the fixture to set up the context with # the tables we need. def test_basic(ctx, database): with pytest.raises(KeyError): ctx.catalog("non-existent") default = ctx.catalog() assert default.names() == {"public"} for db in [default.schema("public"), default.schema()]: assert db.names() == {"csv1", "csv", "csv2"} table = db.table("csv") assert table.kind == "physical" assert table.schema == pa.schema( [ pa.field("int", pa.int64(), nullable=True), pa.field("str", pa.string(), nullable=True), pa.field("float", pa.float64(), nullable=True), ] ) def create_dataset() -> Table: batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) return Table(dataset) class CustomSchemaProvider(dfn.catalog.SchemaProvider): def __init__(self): self.tables = {"table1": create_dataset()} def table_names(self) -> set[str]: return set(self.tables.keys()) def register_table(self, name: str, table: Table): self.tables[name] = table def deregister_table(self, name, cascade: bool = True): del self.tables[name] def table(self, name: str) -> Table | None: return self.tables[name] def table_exist(self, name: str) -> bool: return name in self.tables class CustomErrorSchemaProvider(CustomSchemaProvider): def table(self, name: str) -> Table | None: message = f"{name} is not an acceptable name" raise ValueError(message) class CustomCatalogProvider(dfn.catalog.CatalogProvider): def __init__(self): self.schemas = {"my_schema": CustomSchemaProvider()} def schema_names(self) -> set[str]: return set(self.schemas.keys()) def schema(self, name: str): return self.schemas[name] def register_schema(self, name: str, schema: dfn.catalog.Schema): self.schemas[name] = schema def deregister_schema(self, name, cascade: bool): del self.schemas[name] class CustomCatalogProviderList(dfn.catalog.CatalogProviderList): def __init__(self): self.catalogs = {"my_catalog": CustomCatalogProvider()} def catalog_names(self) -> set[str]: return set(self.catalogs.keys()) def catalog(self, name: str) -> Catalog | None: return self.catalogs[name] def register_catalog( self, name: str, catalog: CatalogProviderExportable | CatalogProvider | Catalog ) -> None: self.catalogs[name] = catalog class CustomTableProviderFactory(dfn.catalog.TableProviderFactory): def create(self, cmd: dfn.expr.CreateExternalTable): assert cmd.name() == "test_table_factory" return create_dataset() def test_python_catalog_provider_list(ctx: SessionContext): ctx.register_catalog_provider_list(CustomCatalogProviderList()) # Ensure `datafusion` catalog does not exist since # we replaced the catalog list assert ctx.catalog_names() == {"my_catalog"} # Ensure registering works ctx.register_catalog_provider("second_catalog", Catalog.memory_catalog()) assert ctx.catalog_names() == {"my_catalog", "second_catalog"} def test_python_catalog_provider(ctx: SessionContext): ctx.register_catalog_provider("my_catalog", CustomCatalogProvider()) # Check the default catalog provider assert ctx.catalog("datafusion").names() == {"public"} my_catalog = ctx.catalog("my_catalog") assert my_catalog.names() == {"my_schema"} my_catalog.register_schema("second_schema", CustomSchemaProvider()) assert my_catalog.schema_names() == {"my_schema", "second_schema"} my_catalog.deregister_schema("my_schema") assert my_catalog.schema_names() == {"second_schema"} def test_in_memory_providers(ctx: SessionContext): catalog = dfn.catalog.Catalog.memory_catalog() ctx.register_catalog_provider("in_mem_catalog", catalog) assert ctx.catalog_names() == {"datafusion", "in_mem_catalog"} schema = dfn.catalog.Schema.memory_schema() catalog.register_schema("in_mem_schema", schema) schema.register_table("my_table", create_dataset()) batches = ctx.sql("select * from in_mem_catalog.in_mem_schema.my_table").collect() assert len(batches) == 1 assert batches[0].column(0) == pa.array([1, 2, 3]) assert batches[0].column(1) == pa.array([4, 5, 6]) def test_python_schema_provider(ctx: SessionContext): catalog = ctx.catalog() catalog.deregister_schema("public") catalog.register_schema("test_schema1", CustomSchemaProvider()) assert catalog.names() == {"test_schema1"} catalog.register_schema("test_schema2", CustomSchemaProvider()) catalog.deregister_schema("test_schema1") assert catalog.names() == {"test_schema2"} def test_python_table_provider(ctx: SessionContext): catalog = ctx.catalog() catalog.register_schema("custom_schema", CustomSchemaProvider()) schema = catalog.schema("custom_schema") assert schema.table_names() == {"table1"} schema.deregister_table("table1") schema.register_table("table2", create_dataset()) assert schema.table_names() == {"table2"} # Use the default schema instead of our custom schema schema = catalog.schema() schema.register_table("table3", create_dataset()) assert schema.table_names() == {"table3"} schema.deregister_table("table3") schema.register_table("table4", create_dataset()) assert schema.table_names() == {"table4"} def test_schema_register_table_with_pyarrow_dataset(ctx: SessionContext): schema = ctx.catalog().schema() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) table_name = "pa_dataset" try: schema.register_table(table_name, dataset) assert table_name in schema.table_names() result = ctx.sql(f"SELECT a, b FROM {table_name}").collect() assert len(result) == 1 assert result[0].column(0) == pa.array([1, 2, 3]) assert result[0].column(1) == pa.array([4, 5, 6]) finally: schema.deregister_table(table_name) def test_exception_not_mangled(ctx: SessionContext): """Test registering all python providers and running a query against them.""" catalog_name = "custom_catalog" schema_name = "custom_schema" ctx.register_catalog_provider(catalog_name, CustomCatalogProvider()) catalog = ctx.catalog(catalog_name) # Clean out previous schemas if they exist so we can start clean for schema_name in catalog.schema_names(): catalog.deregister_schema(schema_name, cascade=False) catalog.register_schema(schema_name, CustomErrorSchemaProvider()) schema = catalog.schema(schema_name) for table_name in schema.table_names(): schema.deregister_table(table_name) schema.register_table("test_table", create_dataset()) with pytest.raises(ValueError, match=r"^test_table is not an acceptable name$"): ctx.sql(f"select * from {catalog_name}.{schema_name}.test_table") def test_in_end_to_end_python_providers(ctx: SessionContext): """Test registering all python providers and running a query against them.""" all_catalog_names = [ "datafusion", "custom_catalog", "in_mem_catalog", ] all_schema_names = [ "custom_schema", "in_mem_schema", ] ctx.register_catalog_provider(all_catalog_names[1], CustomCatalogProvider()) ctx.register_catalog_provider( all_catalog_names[2], dfn.catalog.Catalog.memory_catalog() ) for catalog_name in all_catalog_names: catalog = ctx.catalog(catalog_name) # Clean out previous schemas if they exist so we can start clean for schema_name in catalog.schema_names(): catalog.deregister_schema(schema_name, cascade=False) catalog.register_schema(all_schema_names[0], CustomSchemaProvider()) catalog.register_schema(all_schema_names[1], dfn.catalog.Schema.memory_schema()) for schema_name in all_schema_names: schema = catalog.schema(schema_name) for table_name in schema.table_names(): schema.deregister_table(table_name) schema.register_table("test_table", create_dataset()) for catalog_name in all_catalog_names: for schema_name in all_schema_names: table_full_name = f"{catalog_name}.{schema_name}.test_table" batches = ctx.sql(f"select * from {table_full_name}").collect() assert len(batches) == 1 assert batches[0].column(0) == pa.array([1, 2, 3]) assert batches[0].column(1) == pa.array([4, 5, 6]) def test_register_python_function_as_udtf(ctx: SessionContext): basic_table = Table(ctx.sql("SELECT 3 AS value")) @udtf("my_table_function") def my_table_function_udtf() -> Table: return basic_table ctx.register_udtf(my_table_function_udtf) result = ctx.sql("SELECT * FROM my_table_function()").collect() assert len(result) == 1 assert len(result[0]) == 1 assert len(result[0][0]) == 1 assert result[0][0][0].as_py() == 3 def test_register_python_table_provider_factory(ctx: SessionContext): ctx.register_table_factory("CUSTOM_FACTORY", CustomTableProviderFactory()) ctx.sql(""" CREATE EXTERNAL TABLE test_table_factory STORED AS CUSTOM_FACTORY LOCATION foo; """).collect() result = ctx.sql("SELECT * FROM test_table_factory;").collect() expect = [ pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) ] assert result == expect apache-datafusion-python-54.0.0/python/tests/test_concurrency.py000066400000000000000000000066471521727353300251460ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. from __future__ import annotations from concurrent.futures import ThreadPoolExecutor import pyarrow as pa from datafusion import SessionContext, col, lit from datafusion import functions as f from datafusion.common import SqlSchema def _run_in_threads(fn, count: int = 8) -> None: with ThreadPoolExecutor(max_workers=count) as executor: futures = [executor.submit(fn, i) for i in range(count)] for future in futures: # Propagate any exception raised in the worker thread. future.result() def test_concurrent_access_to_shared_structures() -> None: """Exercise SqlSchema and DataFrame concurrently.""" schema = SqlSchema("concurrency") ctx = SessionContext() batch = pa.record_batch([pa.array([1, 2, 3], type=pa.int32())], names=["value"]) df = ctx.create_dataframe([[batch]]) expected_rows = batch.num_rows def worker(index: int) -> None: schema.name = f"concurrency-{index}" assert schema.name.startswith("concurrency-") # Exercise getters that use internal locks. assert isinstance(schema.tables, list) assert isinstance(schema.views, list) assert isinstance(schema.functions, list) batches = df.collect() assert sum(batch.num_rows for batch in batches) == expected_rows _run_in_threads(worker, count=12) def test_case_builder_reuse_from_multiple_threads() -> None: """Ensure the case builder can be safely reused across threads.""" ctx = SessionContext() values = pa.array([0, 1, 2, 3, 4], type=pa.int32()) df = ctx.create_dataframe([[pa.record_batch([values], names=["value"])]]) base_builder = f.case(col("value")) def add_case(i: int) -> None: nonlocal base_builder base_builder = base_builder.when(lit(i), lit(f"value-{i}")) _run_in_threads(add_case, count=8) with ThreadPoolExecutor(max_workers=2) as executor: otherwise_future = executor.submit(base_builder.otherwise, lit("default")) case_expr = otherwise_future.result() result = df.select(case_expr.alias("label")).collect() assert sum(batch.num_rows for batch in result) == len(values) predicate_builder = f.when(col("value") == lit(0), lit("zero")) def add_predicate(i: int) -> None: predicate_builder.when(col("value") == lit(i + 1), lit(f"value-{i + 1}")) _run_in_threads(add_predicate, count=4) with ThreadPoolExecutor(max_workers=2) as executor: end_future = executor.submit(predicate_builder.end) predicate_expr = end_future.result() result = df.select(predicate_expr.alias("label")).collect() assert sum(batch.num_rows for batch in result) == len(values) apache-datafusion-python-54.0.0/python/tests/test_context.py000066400000000000000000001026551521727353300242740ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import datetime as dt import gzip import pathlib import pyarrow as pa import pyarrow.dataset as ds import pytest from datafusion import ( CsvReadOptions, DataFrame, RuntimeEnvBuilder, SessionConfig, SessionContext, SQLOptions, Table, column, literal, udf, ) def test_create_context_no_args(): SessionContext() def test_create_context_session_config_only(): SessionContext(config=SessionConfig()) def test_create_context_runtime_config_only(): SessionContext(runtime=RuntimeEnvBuilder()) @pytest.mark.parametrize("path_to_str", [True, False]) def test_runtime_configs(tmp_path, path_to_str): path1 = tmp_path / "dir1" path2 = tmp_path / "dir2" path1 = str(path1) if path_to_str else path1 path2 = str(path2) if path_to_str else path2 runtime = RuntimeEnvBuilder().with_disk_manager_specified(path1, path2) config = SessionConfig().with_default_catalog_and_schema("foo", "bar") ctx = SessionContext(config, runtime) assert ctx is not None db = ctx.catalog("foo").schema("bar") assert db is not None @pytest.mark.parametrize("path_to_str", [True, False]) def test_temporary_files(tmp_path, path_to_str): path = str(tmp_path) if path_to_str else tmp_path runtime = RuntimeEnvBuilder().with_temp_file_path(path) config = SessionConfig().with_default_catalog_and_schema("foo", "bar") ctx = SessionContext(config, runtime) assert ctx is not None db = ctx.catalog("foo").schema("bar") assert db is not None def test_create_context_with_all_valid_args(): runtime = RuntimeEnvBuilder().with_disk_manager_os().with_fair_spill_pool(10000000) config = ( SessionConfig() .with_create_default_catalog_and_schema(enabled=True) .with_default_catalog_and_schema("foo", "bar") .with_target_partitions(1) .with_information_schema(enabled=True) .with_repartition_joins(enabled=False) .with_repartition_aggregations(enabled=False) .with_repartition_windows(enabled=False) .with_parquet_pruning(enabled=False) ) ctx = SessionContext(config, runtime) # verify that at least some of the arguments worked ctx.catalog("foo").schema("bar") with pytest.raises(KeyError): ctx.catalog("datafusion") def test_register_record_batches(ctx): # create a RecordBatch and register it as memtable batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) ctx.register_record_batches("t", [[batch]]) assert ctx.catalog().schema().names() == {"t"} result = ctx.sql("SELECT a+b, a-b FROM t").collect() assert result[0].column(0) == pa.array([5, 7, 9]) assert result[0].column(1) == pa.array([-3, -3, -3]) def test_create_dataframe_registers_unique_table_name(ctx): # create a RecordBatch and register it as memtable batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df = ctx.create_dataframe([[batch]]) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert len(tables[0]) == 33 assert tables[0].startswith("c") # ensure that the rest of the table name contains # only hexadecimal numbers for c in tables[0][1:]: assert c in "0123456789abcdef" def test_create_dataframe_registers_with_defined_table_name(ctx): # create a RecordBatch and register it as memtable batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df = ctx.create_dataframe([[batch]], name="tbl") tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert tables[0] == "tbl" def test_from_arrow_table(ctx): # create a PyArrow table data = {"a": [1, 2, 3], "b": [4, 5, 6]} table = pa.Table.from_pydict(data) # convert to DataFrame df = ctx.from_arrow(table) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert df.collect()[0].num_rows == 3 def record_batch_generator(num_batches: int): schema = pa.schema([("a", pa.int64()), ("b", pa.int64())]) for _i in range(num_batches): yield pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], schema=schema ) @pytest.mark.parametrize( "source", [ # __arrow_c_array__ sources pa.array([{"a": 1, "b": 4}, {"a": 2, "b": 5}, {"a": 3, "b": 6}]), # __arrow_c_stream__ sources pa.RecordBatch.from_pydict({"a": [1, 2, 3], "b": [4, 5, 6]}), pa.RecordBatchReader.from_batches( pa.schema([("a", pa.int64()), ("b", pa.int64())]), record_batch_generator(1) ), pa.Table.from_pydict({"a": [1, 2, 3], "b": [4, 5, 6]}), ], ) def test_from_arrow_sources(ctx, source) -> None: df = ctx.from_arrow(source) assert df assert isinstance(df, DataFrame) assert df.schema().names == ["a", "b"] assert df.count() == 3 def test_from_arrow_table_with_name(ctx): # create a PyArrow table data = {"a": [1, 2, 3], "b": [4, 5, 6]} table = pa.Table.from_pydict(data) # convert to DataFrame with optional name df = ctx.from_arrow(table, name="tbl") tables = list(ctx.catalog().schema().names()) assert df assert tables[0] == "tbl" def test_from_arrow_table_empty(ctx): data = {"a": [], "b": []} schema = pa.schema([("a", pa.int32()), ("b", pa.string())]) table = pa.Table.from_pydict(data, schema=schema) # convert to DataFrame df = ctx.from_arrow(table) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert len(df.collect()) == 0 def test_from_arrow_table_empty_no_schema(ctx): data = {"a": [], "b": []} table = pa.Table.from_pydict(data) # convert to DataFrame df = ctx.from_arrow(table) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert len(df.collect()) == 0 def test_from_pylist(ctx): # create a dataframe from Python list data = [ {"a": 1, "b": 4}, {"a": 2, "b": 5}, {"a": 3, "b": 6}, ] df = ctx.from_pylist(data) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert df.collect()[0].num_rows == 3 def test_from_pydict(ctx): # create a dataframe from Python dictionary data = {"a": [1, 2, 3], "b": [4, 5, 6]} df = ctx.from_pydict(data) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert df.collect()[0].num_rows == 3 def test_from_pandas(ctx): # create a dataframe from pandas dataframe pd = pytest.importorskip("pandas") data = {"a": [1, 2, 3], "b": [4, 5, 6]} pandas_df = pd.DataFrame(data) df = ctx.from_pandas(pandas_df) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert df.collect()[0].num_rows == 3 def test_from_polars(ctx): # create a dataframe from Polars dataframe pd = pytest.importorskip("polars") data = {"a": [1, 2, 3], "b": [4, 5, 6]} polars_df = pd.DataFrame(data) df = ctx.from_polars(polars_df) tables = list(ctx.catalog().schema().names()) assert df assert len(tables) == 1 assert isinstance(df, DataFrame) assert set(df.schema().names) == {"a", "b"} assert df.collect()[0].num_rows == 3 def test_register_table(ctx, database): default = ctx.catalog() public = default.schema("public") assert public.names() == {"csv", "csv1", "csv2"} table = public.table("csv") ctx.register_table("csv3", table) assert public.names() == {"csv", "csv1", "csv2", "csv3"} def test_read_table_from_catalog(ctx, database): default = ctx.catalog() public = default.schema("public") assert public.names() == {"csv", "csv1", "csv2"} table = public.table("csv") table_df = ctx.read_table(table) table_df.show() def test_read_table_from_df(ctx): df = ctx.from_pydict({"a": [1, 2]}) result = ctx.read_table(df).collect() assert [b.to_pydict() for b in result] == [{"a": [1, 2]}] def test_read_table_from_dataset(ctx): batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) result = ctx.read_table(dataset).collect() assert result[0].column(0) == pa.array([1, 2, 3]) assert result[0].column(1) == pa.array([4, 5, 6]) def test_deregister_table(ctx, database): default = ctx.catalog() public = default.schema("public") assert public.names() == {"csv", "csv1", "csv2"} ctx.deregister_table("csv") assert public.names() == {"csv1", "csv2"} def test_deregister_udf(): ctx = SessionContext() is_null = udf( lambda x: x.is_null(), [pa.float64()], pa.bool_(), volatility="immutable", name="my_is_null", ) ctx.register_udf(is_null) # Verify it works df = ctx.from_pydict({"a": [1.0, None]}) ctx.register_table("t", df.into_view()) result = ctx.sql("SELECT my_is_null(a) FROM t").collect() assert result[0].column(0) == pa.array([False, True]) # Deregister and verify it's gone ctx.deregister_udf("my_is_null") with pytest.raises(ValueError): ctx.sql("SELECT my_is_null(a) FROM t").collect() def test_deregister_udaf(): import pyarrow.compute as pc ctx = SessionContext() from datafusion import Accumulator, udaf class MySum(Accumulator): def __init__(self): self._sum = 0.0 def update(self, values: pa.Array) -> None: self._sum += pc.sum(values).as_py() def merge(self, states: list[pa.Array]) -> None: self._sum += pc.sum(states[0]).as_py() def state(self) -> list: return [self._sum] def evaluate(self) -> pa.Scalar: return self._sum my_sum = udaf( MySum, [pa.float64()], pa.float64(), [pa.float64()], volatility="immutable", name="my_sum", ) ctx.register_udaf(my_sum) df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) ctx.register_table("t", df.into_view()) result = ctx.sql("SELECT my_sum(a) FROM t").collect() assert result[0].column(0) == pa.array([6.0]) ctx.deregister_udaf("my_sum") with pytest.raises(ValueError): ctx.sql("SELECT my_sum(a) FROM t").collect() def test_deregister_udwf(): ctx = SessionContext() from datafusion import udwf from datafusion.user_defined import WindowEvaluator class MyRowNumber(WindowEvaluator): def __init__(self): self._row = 0 def evaluate_all(self, values, num_rows): return pa.array(list(range(1, num_rows + 1)), type=pa.uint64()) my_row_number = udwf( MyRowNumber, [pa.float64()], pa.uint64(), volatility="immutable", name="my_row_number", ) ctx.register_udwf(my_row_number) df = ctx.from_pydict({"a": [1.0, 2.0, 3.0]}) ctx.register_table("t", df.into_view()) result = ctx.sql("SELECT my_row_number(a) OVER () FROM t").collect() assert result[0].column(0) == pa.array([1, 2, 3], type=pa.uint64()) ctx.deregister_udwf("my_row_number") with pytest.raises(ValueError): ctx.sql("SELECT my_row_number(a) OVER () FROM t").collect() def test_deregister_udtf(): import pyarrow.dataset as ds ctx = SessionContext() from datafusion import Table, udtf class MyTable: def __call__(self): batch = pa.RecordBatch.from_pydict({"x": [1, 2, 3]}) return Table(ds.dataset([batch])) my_table = udtf(MyTable(), "my_table") ctx.register_udtf(my_table) result = ctx.sql("SELECT * FROM my_table()").collect() assert result[0].column(0) == pa.array([1, 2, 3]) ctx.deregister_udtf("my_table") with pytest.raises(ValueError): ctx.sql("SELECT * FROM my_table()").collect() def test_register_table_from_dataframe(ctx): df = ctx.from_pydict({"a": [1, 2]}) ctx.register_table("df_tbl", df) result = ctx.sql("SELECT * FROM df_tbl").collect() assert [b.to_pydict() for b in result] == [{"a": [1, 2]}] @pytest.mark.parametrize("temporary", [True, False]) def test_register_table_from_dataframe_into_view(ctx, temporary): df = ctx.from_pydict({"a": [1, 2]}) table = df.into_view(temporary=temporary) assert isinstance(table, Table) if temporary: assert table.kind == "temporary" else: assert table.kind == "view" ctx.register_table("view_tbl", table) result = ctx.sql("SELECT * FROM view_tbl").collect() assert [b.to_pydict() for b in result] == [{"a": [1, 2]}] def test_table_from_dataframe(ctx): df = ctx.from_pydict({"a": [1, 2]}) table = Table(df) assert isinstance(table, Table) ctx.register_table("from_dataframe_tbl", table) result = ctx.sql("SELECT * FROM from_dataframe_tbl").collect() assert [b.to_pydict() for b in result] == [{"a": [1, 2]}] def test_table_from_dataframe_internal(ctx): df = ctx.from_pydict({"a": [1, 2]}) table = Table(df.df) assert isinstance(table, Table) ctx.register_table("from_internal_dataframe_tbl", table) result = ctx.sql("SELECT * FROM from_internal_dataframe_tbl").collect() assert [b.to_pydict() for b in result] == [{"a": [1, 2]}] def test_register_dataset(ctx): # create a RecordBatch and register it as a pyarrow.dataset.Dataset batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) ctx.register_dataset("t", dataset) assert ctx.catalog().schema().names() == {"t"} result = ctx.sql("SELECT a+b, a-b FROM t").collect() assert result[0].column(0) == pa.array([5, 7, 9]) assert result[0].column(1) == pa.array([-3, -3, -3]) def test_dataset_filter(ctx, capfd): # create a RecordBatch and register it as a pyarrow.dataset.Dataset batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) ctx.register_dataset("t", dataset) assert ctx.catalog().schema().names() == {"t"} df = ctx.sql("SELECT a+b, a-b FROM t WHERE a BETWEEN 2 and 3 AND b > 5") # Make sure the filter was pushed down in Physical Plan df.explain() captured = capfd.readouterr() assert "filter_expr=(((a >= 2) and (a <= 3)) and (b > 5))" in captured.out result = df.collect() assert result[0].column(0) == pa.array([9]) assert result[0].column(1) == pa.array([-3]) def test_dataset_count(ctx): # `datafusion-python` issue: https://github.com/apache/datafusion-python/issues/800 batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) ctx.register_dataset("t", dataset) # Testing the dataframe API df = ctx.table("t") assert df.count() == 3 # Testing the SQL API count = ctx.sql("SELECT COUNT(*) FROM t") count = count.collect() assert count[0].column(0) == pa.array([3]) def test_pyarrow_predicate_pushdown_is_null(ctx, capfd): """Ensure that pyarrow filter gets pushed down for `IsNull`""" # create a RecordBatch and register it as a pyarrow.dataset.Dataset batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6]), pa.array([7, None, 9])], names=["a", "b", "c"], ) dataset = ds.dataset([batch]) ctx.register_dataset("t", dataset) # Make sure the filter was pushed down in Physical Plan df = ctx.sql("SELECT a FROM t WHERE c is NULL") df.explain() captured = capfd.readouterr() assert "filter_expr=is_null(c, {nan_is_null=false})" in captured.out result = df.collect() assert result[0].column(0) == pa.array([2]) def test_pyarrow_predicate_pushdown_timestamp(ctx, tmpdir, capfd): """Ensure that pyarrow filter gets pushed down for timestamp""" # Ref: https://github.com/apache/datafusion-python/issues/703 # create pyarrow dataset with no actual files col_type = pa.timestamp("ns", "+00:00") nyd_2000 = pa.scalar(dt.datetime(2000, 1, 1, tzinfo=dt.timezone.utc), col_type) pa_dataset_fs = pa.fs.SubTreeFileSystem(str(tmpdir), pa.fs.LocalFileSystem()) pa_dataset_format = pa.dataset.ParquetFileFormat() pa_dataset_partition = pa.dataset.field("a") <= nyd_2000 fragments = [ # NOTE: we never actually make this file. # Working predicate pushdown means it never gets accessed pa_dataset_format.make_fragment( "1.parquet", filesystem=pa_dataset_fs, partition_expression=pa_dataset_partition, ) ] pa_dataset = pa.dataset.FileSystemDataset( fragments, pa.schema([pa.field("a", col_type)]), pa_dataset_format, pa_dataset_fs, ) ctx.register_dataset("t", pa_dataset) # the partition for our only fragment is for a < 2000-01-01. # so querying for a > 2024-01-01 should not touch any files df = ctx.sql("SELECT * FROM t WHERE a > '2024-01-01T00:00:00+00:00'") assert df.collect() == [] def test_dataset_filter_nested_data(ctx): # create Arrow StructArrays to test nested data types data = pa.StructArray.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) batch = pa.RecordBatch.from_arrays( [data], names=["nested_data"], ) dataset = ds.dataset([batch]) ctx.register_dataset("t", dataset) assert ctx.catalog().schema().names() == {"t"} df = ctx.table("t") # This filter will not be pushed down to DatasetExec since it # isn't supported df = df.filter(column("nested_data")["b"] > literal(5)).select( column("nested_data")["a"] + column("nested_data")["b"], column("nested_data")["a"] - column("nested_data")["b"], ) result = df.collect() assert result[0].column(0) == pa.array([9]) assert result[0].column(1) == pa.array([-3]) def test_table_exist(ctx): batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) ctx.register_dataset("t", dataset) assert ctx.table_exist("t") is True def test_table_not_found(ctx): from uuid import uuid4 with pytest.raises(KeyError): ctx.table(f"not-found-{uuid4()}") def test_session_start_time(ctx): import datetime import re st = ctx.session_start_time() assert isinstance(st, str) # Truncate nanoseconds to microseconds for Python 3.10 compat st = re.sub(r"(\.\d{6})\d+", r"\1", st) dt = datetime.datetime.fromisoformat(st) assert dt.isoformat() def test_enable_ident_normalization(ctx): assert ctx.enable_ident_normalization() is True ctx.sql("SET datafusion.sql_parser.enable_ident_normalization = false") assert ctx.enable_ident_normalization() is False def test_parse_sql_expr(ctx): from datafusion.common import DFSchema schema = DFSchema.empty() expr = ctx.parse_sql_expr("1 + 2", schema) assert str(expr) == "Expr(Int64(1) + Int64(2))" def test_execute_logical_plan(ctx): df = ctx.from_pydict({"a": [1, 2, 3]}) plan = df.logical_plan() df2 = ctx.execute_logical_plan(plan) result = df2.collect() assert result[0].column(0) == pa.array([1, 2, 3]) def test_refresh_catalogs(ctx): ctx.refresh_catalogs() def test_remove_optimizer_rule(ctx): assert ctx.remove_optimizer_rule("push_down_filter") is True assert ctx.remove_optimizer_rule("nonexistent_rule") is False def test_table_provider(ctx): batch = pa.RecordBatch.from_pydict({"x": [10, 20, 30]}) ctx.register_record_batches("provider_test", [[batch]]) tbl = ctx.table_provider("provider_test") assert tbl.schema == pa.schema([("x", pa.int64())]) def test_table_provider_not_found(ctx): with pytest.raises(KeyError): ctx.table_provider("nonexistent_table") def test_read_json(ctx): path = pathlib.Path(__file__).parent.resolve() # Default test_data_path = path / "data_test_context" / "data.json" df = ctx.read_json(test_data_path) result = df.collect() assert result[0].column(0) == pa.array(["a", "b", "c"]) assert result[0].column(1) == pa.array([1, 2, 3]) # Schema schema = pa.schema( [ pa.field("A", pa.string(), nullable=True), ] ) df = ctx.read_json(test_data_path, schema=schema) result = df.collect() assert result[0].column(0) == pa.array(["a", "b", "c"]) assert result[0].schema == schema # File extension test_data_path = path / "data_test_context" / "data.json" df = ctx.read_json(test_data_path, file_extension=".json") result = df.collect() assert result[0].column(0) == pa.array(["a", "b", "c"]) assert result[0].column(1) == pa.array([1, 2, 3]) def test_read_json_compressed(ctx, tmp_path): path = pathlib.Path(__file__).parent.resolve() test_data_path = path / "data_test_context" / "data.json" # File compression type gzip_path = tmp_path / "data.json.gz" with ( pathlib.Path.open(test_data_path, "rb") as csv_file, gzip.open(gzip_path, "wb") as gzipped_file, ): gzipped_file.writelines(csv_file) df = ctx.read_json(gzip_path, file_extension=".gz", file_compression_type="gz") result = df.collect() assert result[0].column(0) == pa.array(["a", "b", "c"]) assert result[0].column(1) == pa.array([1, 2, 3]) def test_read_csv(ctx): csv_df = ctx.read_csv(path="testing/data/csv/aggregate_test_100.csv") csv_df.select(column("c1")).show() def test_read_csv_list(ctx): csv_df = ctx.read_csv(path=["testing/data/csv/aggregate_test_100.csv"]) expected = csv_df.count() * 2 double_csv_df = ctx.read_csv( path=[ "testing/data/csv/aggregate_test_100.csv", "testing/data/csv/aggregate_test_100.csv", ] ) actual = double_csv_df.count() double_csv_df.select(column("c1")).show() assert actual == expected def test_read_csv_compressed(ctx, tmp_path): test_data_path = pathlib.Path("testing/data/csv/aggregate_test_100.csv") expected = ctx.read_csv(test_data_path).collect() # File compression type gzip_path = tmp_path / "aggregate_test_100.csv.gz" with ( pathlib.Path.open(test_data_path, "rb") as csv_file, gzip.open(gzip_path, "wb") as gzipped_file, ): gzipped_file.writelines(csv_file) csv_df = ctx.read_csv(gzip_path, file_extension=".gz", file_compression_type="gz") assert csv_df.collect() == expected csv_df = ctx.read_csv( gzip_path, options=CsvReadOptions(file_extension=".gz", file_compression_type="gz"), ) assert csv_df.collect() == expected def test_read_parquet(ctx): parquet_df = ctx.read_parquet(path="parquet/data/alltypes_plain.parquet") parquet_df.show() assert parquet_df is not None path = pathlib.Path.cwd() / "parquet/data/alltypes_plain.parquet" parquet_df = ctx.read_parquet(path=path) assert parquet_df is not None def test_read_avro(ctx): avro_df = ctx.read_avro(path="testing/data/avro/alltypes_plain.avro") avro_df.show() assert avro_df is not None path = pathlib.Path.cwd() / "testing/data/avro/alltypes_plain.avro" avro_df = ctx.read_avro(path=path) assert avro_df is not None def test_read_arrow(ctx, tmp_path): # Write an Arrow IPC file, then read it back table = pa.table({"a": [1, 2, 3], "b": ["x", "y", "z"]}) arrow_path = tmp_path / "test.arrow" with pa.ipc.new_file(str(arrow_path), table.schema) as writer: writer.write_table(table) df = ctx.read_arrow(str(arrow_path)) result = df.collect() assert result[0].column(0) == pa.array([1, 2, 3]) assert result[0].column(1) == pa.array(["x", "y", "z"]) # Also verify pathlib.Path works df = ctx.read_arrow(arrow_path) result = df.collect() assert result[0].column(0) == pa.array([1, 2, 3]) def test_read_empty(ctx): df = ctx.read_empty() result = df.collect() assert len(result) == 1 assert result[0].num_columns == 0 df = ctx.empty_table() result = df.collect() assert len(result) == 1 assert result[0].num_columns == 0 def test_register_arrow(ctx, tmp_path): # Write an Arrow IPC file, then register and query it table = pa.table({"x": [10, 20, 30]}) arrow_path = tmp_path / "test.arrow" with pa.ipc.new_file(str(arrow_path), table.schema) as writer: writer.write_table(table) ctx.register_arrow("arrow_tbl", str(arrow_path)) result = ctx.sql("SELECT * FROM arrow_tbl").collect() assert result[0].column(0) == pa.array([10, 20, 30]) # Also verify pathlib.Path works ctx.register_arrow("arrow_tbl_path", arrow_path) result = ctx.sql("SELECT * FROM arrow_tbl_path").collect() assert result[0].column(0) == pa.array([10, 20, 30]) def test_register_batch(ctx): batch = pa.RecordBatch.from_pydict({"a": [1, 2, 3], "b": [4, 5, 6]}) ctx.register_batch("batch_tbl", batch) result = ctx.sql("SELECT * FROM batch_tbl").collect() assert result[0].column(0) == pa.array([1, 2, 3]) assert result[0].column(1) == pa.array([4, 5, 6]) def test_register_batch_empty(ctx): batch = pa.RecordBatch.from_pydict({"a": pa.array([], type=pa.int64())}) ctx.register_batch("empty_batch_tbl", batch) result = ctx.sql("SELECT * FROM empty_batch_tbl").collect() assert result[0].num_rows == 0 def test_read_batch_returns_dataframe(ctx): batch = pa.RecordBatch.from_pydict({"a": [1, 2, 3], "b": [4, 5, 6]}) df = ctx.read_batch(batch) assert df.to_pydict() == {"a": [1, 2, 3], "b": [4, 5, 6]} # read_batch should not register a named table. assert ctx.catalog().schema().names() == set() def test_read_batches_concatenates(ctx): b1 = pa.RecordBatch.from_pydict({"a": [1, 2]}) b2 = pa.RecordBatch.from_pydict({"a": [3, 4]}) df = ctx.read_batches([b1, b2]) assert df.to_pydict() == {"a": [1, 2, 3, 4]} def test_read_batches_accepts_iterable(ctx): b1 = pa.RecordBatch.from_pydict({"a": [1, 2]}) b2 = pa.RecordBatch.from_pydict({"a": [3, 4]}) # Generator: ensures non-list iterables are materialized before FFI. df = ctx.read_batches(b for b in (b1, b2)) assert df.to_pydict() == {"a": [1, 2, 3, 4]} # Tuple: same. df = ctx.read_batches((b1, b2)) assert df.to_pydict() == {"a": [1, 2, 3, 4]} def test_create_sql_options(): SQLOptions() def test_sql_with_options_no_ddl(ctx): sql = "CREATE TABLE IF NOT EXISTS valuetable AS VALUES(1,'HELLO'),(12,'DATAFUSION')" ctx.sql(sql) options = SQLOptions().with_allow_ddl(allow=False) with pytest.raises(Exception, match="DDL"): ctx.sql_with_options(sql, options=options) def test_sql_with_options_no_dml(ctx): table_name = "t" batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) dataset = ds.dataset([batch]) ctx.register_dataset(table_name, dataset) sql = f'INSERT INTO "{table_name}" VALUES (1, 2), (2, 3);' ctx.sql(sql) options = SQLOptions().with_allow_dml(allow=False) with pytest.raises(Exception, match="DML"): ctx.sql_with_options(sql, options=options) def test_sql_with_options_no_statements(ctx): sql = "SET time zone = 1;" ctx.sql(sql) options = SQLOptions().with_allow_statements(allow=False) with pytest.raises(Exception, match="SetVariable"): ctx.sql_with_options(sql, options=options) @pytest.fixture def batch(): return pa.RecordBatch.from_arrays( [pa.array([4, 5, 6])], names=["a"], ) def test_create_dataframe_with_global_ctx(batch): ctx = SessionContext.global_ctx() df = ctx.create_dataframe([[batch]]) result = df.collect()[0].column(0) assert result == pa.array([4, 5, 6]) def test_csv_read_options_builder_pattern(): """Test CsvReadOptions builder pattern.""" from datafusion import CsvReadOptions options = ( CsvReadOptions() .with_has_header(False) .with_delimiter("|") .with_quote("'") .with_schema_infer_max_records(2000) .with_truncated_rows(True) .with_newlines_in_values(True) .with_file_extension(".tsv") ) assert options.has_header is False assert options.delimiter == "|" assert options.quote == "'" assert options.schema_infer_max_records == 2000 assert options.truncated_rows is True assert options.newlines_in_values is True assert options.file_extension == ".tsv" def read_csv_with_options_inner( tmp_path: pathlib.Path, csv_content: str, options: CsvReadOptions, expected: pa.RecordBatch, as_read: bool, global_ctx: bool, ) -> None: from datafusion import SessionContext # Create a test CSV file group_dir = tmp_path / "group=a" group_dir.mkdir(exist_ok=True) csv_path = group_dir / "test.csv" csv_path.write_text(csv_content, newline="\n") ctx = SessionContext() if as_read: if global_ctx: from datafusion.io import read_csv df = read_csv(str(tmp_path), options=options) else: df = ctx.read_csv(str(tmp_path), options=options) else: ctx.register_csv("test_table", str(tmp_path), options=options) df = ctx.sql("SELECT * FROM test_table") df.show() # Verify the data result = df.collect() assert len(result) == 1 assert result[0] == expected @pytest.mark.parametrize( ("as_read", "global_ctx"), [ (True, True), (True, False), (False, False), ], ) def test_read_csv_with_options(tmp_path, as_read, global_ctx): """Test reading CSV with CsvReadOptions.""" csv_content = "Alice;30;|New York; NY|\nBob;25\n#Charlie;35;Paris\nPhil;75;Detroit' MI\nKarin;50;|Stockholm\nSweden|" # noqa: E501 # Some of the read options are difficult to test in combination # such as schema and schema_infer_max_records so run multiple tests # file_sort_order doesn't impact reading, but included here to ensure # all options parse correctly options = CsvReadOptions( has_header=False, delimiter=";", quote="|", terminator="\n", escape="\\", comment="#", newlines_in_values=True, schema_infer_max_records=1, null_regex="[pP]+aris", truncated_rows=True, file_sort_order=[[column("column_1").sort(), column("column_2")], ["column_3"]], ) expected = pa.RecordBatch.from_arrays( [ pa.array(["Alice", "Bob", "Phil", "Karin"]), pa.array([30, 25, 75, 50]), pa.array(["New York; NY", None, "Detroit' MI", "Stockholm\nSweden"]), ], names=["column_1", "column_2", "column_3"], ) read_csv_with_options_inner( tmp_path, csv_content, options, expected, as_read, global_ctx ) schema = pa.schema( [ pa.field("name", pa.string(), nullable=False), pa.field("age", pa.float32(), nullable=False), pa.field("location", pa.string(), nullable=True), ] ) options.with_schema(schema) expected = pa.RecordBatch.from_arrays( [ pa.array(["Alice", "Bob", "Phil", "Karin"]), pa.array([30.0, 25.0, 75.0, 50.0]), pa.array(["New York; NY", None, "Detroit' MI", "Stockholm\nSweden"]), ], schema=schema, ) read_csv_with_options_inner( tmp_path, csv_content, options, expected, as_read, global_ctx ) csv_content = "name,age\nAlice,30\nBob,25\nCharlie,35\nDiego,40\nEmily,15" expected = pa.RecordBatch.from_arrays( [ pa.array(["Alice", "Bob", "Charlie", "Diego", "Emily"]), pa.array([30, 25, 35, 40, 15]), pa.array(["a", "a", "a", "a", "a"]), ], schema=pa.schema( [ pa.field("name", pa.string(), nullable=True), pa.field("age", pa.int64(), nullable=True), pa.field("group", pa.string(), nullable=False), ] ), ) options = CsvReadOptions( table_partition_cols=[("group", pa.string())], ) read_csv_with_options_inner( tmp_path, csv_content, options, expected, as_read, global_ctx ) apache-datafusion-python-54.0.0/python/tests/test_dataframe.py000066400000000000000000003735721521727353300245440ustar00rootroot00000000000000# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, # software distributed under the License is distributed on an # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY # KIND, either express or implied. See the License for the # specific language governing permissions and limitations # under the License. import ctypes import datetime import itertools import os import re import threading import time from pathlib import Path from typing import Any import pyarrow as pa import pyarrow.parquet as pq import pytest from datafusion import ( DataFrame, ExplainFormat, InsertOp, ParquetColumnOptions, ParquetWriterOptions, RecordBatch, SessionContext, WindowFrame, column, literal, udf, ) from datafusion import ( col as df_col, ) from datafusion import ( functions as f, ) from datafusion.dataframe import DataFrameWriteOptions from datafusion.dataframe_formatter import ( DataFrameHtmlFormatter, configure_formatter, get_formatter, reset_formatter, ) from datafusion.expr import EXPR_TYPE_ERROR, Window from pyarrow.csv import write_csv pa_cffi = pytest.importorskip("pyarrow.cffi") MB = 1024 * 1024 @pytest.fixture def ctx(): return SessionContext() @pytest.fixture def df(ctx): # create a RecordBatch and a new DataFrame from it batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6]), pa.array([8, 5, 8])], names=["a", "b", "c"], ) return ctx.from_arrow(batch) @pytest.fixture def large_df(): ctx = SessionContext() rows = 100000 data = { "a": list(range(rows)), "b": [f"s-{i}" for i in range(rows)], "c": [float(i + 0.1) for i in range(rows)], } batch = pa.record_batch(data) return ctx.from_arrow(batch) @pytest.fixture def large_multi_batch_df(): """Create a DataFrame with multiple record batches for testing stream behavior. This fixture creates 10 batches of 10,000 rows each (100,000 rows total), ensuring the DataFrame spans multiple batches. This is essential for testing that memory limits actually cause early stream termination rather than truncating all collected data. """ ctx = SessionContext() # Create multiple batches, each with 10,000 rows batches = [] rows_per_batch = 10000 num_batches = 10 for batch_idx in range(num_batches): start_row = batch_idx * rows_per_batch end_row = start_row + rows_per_batch data = { "a": list(range(start_row, end_row)), "b": [f"s-{i}" for i in range(start_row, end_row)], "c": [float(i + 0.1) for i in range(start_row, end_row)], } batch = pa.record_batch(data) batches.append(batch) # Register as record batches to maintain multi-batch structure # Using [batches] wraps list in another list as required by register_record_batches ctx.register_record_batches("large_multi_batch_data", [batches]) return ctx.table("large_multi_batch_data") @pytest.fixture def struct_df(): ctx = SessionContext() # create a RecordBatch and a new DataFrame from it batch = pa.RecordBatch.from_arrays( [pa.array([{"c": 1}, {"c": 2}, {"c": 3}]), pa.array([4, 5, 6])], names=["a", "b"], ) return ctx.create_dataframe([[batch]]) @pytest.fixture def nested_df(): ctx = SessionContext() # create a RecordBatch and a new DataFrame from it # Intentionally make each array of different length batch = pa.RecordBatch.from_arrays( [pa.array([[1], [2, 3], [4, 5, 6], None]), pa.array([7, 8, 9, 10])], names=["a", "b"], ) return ctx.create_dataframe([[batch]]) @pytest.fixture def aggregate_df(): ctx = SessionContext() ctx.register_csv("test", "testing/data/csv/aggregate_test_100.csv") return ctx.sql("select c1, sum(c2) from test group by c1") @pytest.fixture def partitioned_df(): ctx = SessionContext() # create a RecordBatch and a new DataFrame from it batch = pa.RecordBatch.from_arrays( [ pa.array([0, 1, 2, 3, 4, 5, 6]), pa.array([7, None, 7, 8, 9, None, 9]), pa.array(["A", "A", "A", "A", "B", "B", "B"]), ], names=["a", "b", "c"], ) return ctx.create_dataframe([[batch]]) @pytest.fixture def clean_formatter_state(): """Reset the HTML formatter after each test.""" reset_formatter() @pytest.fixture def null_df(): """Create a DataFrame with null values of different types.""" ctx = SessionContext() # Create a RecordBatch with nulls across different types batch = pa.RecordBatch.from_arrays( [ pa.array([1, None, 3, None], type=pa.int64()), pa.array([4.5, 6.7, None, None], type=pa.float64()), pa.array(["a", None, "c", None], type=pa.string()), pa.array([True, None, False, None], type=pa.bool_()), pa.array( [10957, None, 18993, None], type=pa.date32() ), # 2000-01-01, null, 2022-01-01, null pa.array( [946684800000, None, 1640995200000, None], type=pa.date64() ), # 2000-01-01, null, 2022-01-01, null ], names=[ "int_col", "float_col", "str_col", "bool_col", "date32_col", "date64_col", ], ) return ctx.create_dataframe([[batch]]) # custom style for testing with html formatter class CustomStyleProvider: def get_cell_style(self) -> str: return ( "background-color: #f5f5f5; color: #333; padding: 8px; border: " "1px solid #ddd;" ) def get_header_style(self) -> str: return ( "background-color: #4285f4; color: white; font-weight: bold; " "padding: 10px; border: 1px solid #3367d6;" ) def count_table_rows(html_content: str) -> int: """Count the number of table rows in HTML content. Args: html_content: HTML string to analyze Returns: Number of table rows found (number of tags) """ return len(re.findall(r" literal(2)).select( column("a") + column("b"), column("a") - column("b"), ) # execute and collect the first (and only) batch result = df1.collect()[0] assert result.column(0) == pa.array([9]) assert result.column(1) == pa.array([-3]) df.show() # verify that if there is no filter applied, internal dataframe is unchanged df2 = df.filter() assert df.df == df2.df df3 = df.filter(column("a") > literal(1), column("b") != literal(6)) result = df3.collect()[0] assert result.column(0) == pa.array([2]) assert result.column(1) == pa.array([5]) assert result.column(2) == pa.array([5]) def test_filter_string_predicates(df): df_str = df.filter("a > 2") result = df_str.collect()[0] assert result.column(0) == pa.array([3]) assert result.column(1) == pa.array([6]) assert result.column(2) == pa.array([8]) df_mixed = df.filter("a > 1", column("b") != literal(6)) result_mixed = df_mixed.collect()[0] assert result_mixed.column(0) == pa.array([2]) assert result_mixed.column(1) == pa.array([5]) assert result_mixed.column(2) == pa.array([5]) df_strings = df.filter("a > 1", "b < 6") result_strings = df_strings.collect()[0] assert result_strings.column(0) == pa.array([2]) assert result_strings.column(1) == pa.array([5]) assert result_strings.column(2) == pa.array([5]) def test_parse_sql_expr(df): plan1 = df.filter(df.parse_sql_expr("a > 2")).logical_plan() plan2 = df.filter(column("a") > literal(2)).logical_plan() # object equality not implemented but string representation should match assert str(plan1) == str(plan2) df1 = df.filter(df.parse_sql_expr("a > 2")).select( column("a") + column("b"), column("a") - column("b"), ) # execute and collect the first (and only) batch result = df1.collect()[0] assert result.column(0) == pa.array([9]) assert result.column(1) == pa.array([-3]) df.show() # verify that if there is no filter applied, internal dataframe is unchanged df2 = df.filter() assert df.df == df2.df df3 = df.filter(df.parse_sql_expr("a > 1"), df.parse_sql_expr("b != 6")) result = df3.collect()[0] assert result.column(0) == pa.array([2]) assert result.column(1) == pa.array([5]) assert result.column(2) == pa.array([5]) def test_show_empty(df, capsys): df_empty = df.filter(column("a") > literal(3)) df_empty.show() captured = capsys.readouterr() assert "DataFrame has no rows" in captured.out def test_show_on_explain(ctx, capsys): ctx.sql("explain select 1").show() captured = capsys.readouterr() assert "1 as Int64(1)" in captured.out ctx.sql("explain analyze select 1").show() captured = capsys.readouterr() assert "1 as Int64(1)" in captured.out def test_sort(df): df = df.sort(column("b").sort(ascending=False)) table = pa.Table.from_batches(df.collect()) expected = {"a": [3, 2, 1], "b": [6, 5, 4], "c": [8, 5, 8]} assert table.to_pydict() == expected def test_sort_string_and_expression_equivalent(df): from datafusion import col result_str = df.sort("a").to_pydict() result_expr = df.sort(col("a")).to_pydict() assert result_str == result_expr def test_sort_unsupported(df): with pytest.raises( TypeError, match=f"Expected Expr or column name.*{re.escape(EXPR_TYPE_ERROR)}", ): df.sort(1) def test_aggregate_string_and_expression_equivalent(df): from datafusion import col result_str = df.aggregate("a", [f.count()]).sort("a").to_pydict() result_expr = df.aggregate(col("a"), [f.count()]).sort("a").to_pydict() assert result_str == result_expr def test_aggregate_tuple_group_by(df): result_list = df.aggregate(["a"], [f.count()]).sort("a").to_pydict() result_tuple = df.aggregate(("a",), [f.count()]).sort("a").to_pydict() assert result_tuple == result_list def test_aggregate_none_group_by_equivalent_to_empty_list(df): result_none = df.aggregate(None, [f.count()]).to_pydict() result_empty = df.aggregate([], [f.count()]).to_pydict() assert result_none == result_empty def test_aggregate_tuple_aggs(df): result_list = df.aggregate("a", [f.count()]).sort("a").to_pydict() result_tuple = df.aggregate("a", (f.count(),)).sort("a").to_pydict() assert result_tuple == result_list def test_filter_string_equivalent(df): df1 = df.filter("a > 1").to_pydict() df2 = df.filter(column("a") > literal(1)).to_pydict() assert df1 == df2 def test_filter_string_invalid(df): with pytest.raises(Exception) as excinfo: df.filter("this is not valid sql").collect() assert "Expected Expr" not in str(excinfo.value) def test_drop(df): df = df.drop("c") # execute and collect the first (and only) batch result = df.collect()[0] assert df.schema().names == ["a", "b"] assert result.column(0) == pa.array([1, 2, 3]) assert result.column(1) == pa.array([4, 5, 6]) def test_limit(df): df = df.limit(1) # execute and collect the first (and only) batch result = df.collect()[0] assert len(result.column(0)) == 1 assert len(result.column(1)) == 1 def test_limit_with_offset(df): # only 3 rows, but limit past the end to ensure that offset is working df = df.limit(5, offset=2) # execute and collect the first (and only) batch result = df.collect()[0] assert len(result.column(0)) == 1 assert len(result.column(1)) == 1 def test_head(df): df = df.head(1) # execute and collect the first (and only) batch result = df.collect()[0] assert result.column(0) == pa.array([1]) assert result.column(1) == pa.array([4]) assert result.column(2) == pa.array([8]) def test_tail(df): df = df.tail(1) # execute and collect the first (and only) batch result = df.collect()[0] assert result.column(0) == pa.array([3]) assert result.column(1) == pa.array([6]) assert result.column(2) == pa.array([8]) def test_with_column_sql_expression(df): df = df.with_column("c", "a + b") # execute and collect the first (and only) batch result = df.collect()[0] assert result.schema.field(0).name == "a" assert result.schema.field(1).name == "b" assert result.schema.field(2).name == "c" assert result.column(0) == pa.array([1, 2, 3]) assert result.column(1) == pa.array([4, 5, 6]) assert result.column(2) == pa.array([5, 7, 9]) def test_with_column(df): df = df.with_column("c", column("a") + column("b")) # execute and collect the first (and only) batch result = df.collect()[0] assert result.schema.field(0).name == "a" assert result.schema.field(1).name == "b" assert result.schema.field(2).name == "c" assert result.column(0) == pa.array([1, 2, 3]) assert result.column(1) == pa.array([4, 5, 6]) assert result.column(2) == pa.array([5, 7, 9]) def test_with_columns(df): df = df.with_columns( (column("a") + column("b")).alias("c"), (column("a") + column("b")).alias("d"), [ (column("a") + column("b")).alias("e"), (column("a") + column("b")).alias("f"), ], g=(column("a") + column("b")), ) # execute and collect the first (and only) batch result = df.collect()[0] assert result.schema.field(0).name == "a" assert result.schema.field(1).name == "b" assert result.schema.field(2).name == "c" assert result.schema.field(3).name == "d" assert result.schema.field(4).name == "e" assert result.schema.field(5).name == "f" assert result.schema.field(6).name == "g" assert result.column(0) == pa.array([1, 2, 3]) assert result.column(1) == pa.array([4, 5, 6]) assert result.column(2) == pa.array([5, 7, 9]) assert result.column(3) == pa.array([5, 7, 9]) assert result.column(4) == pa.array([5, 7, 9]) assert result.column(5) == pa.array([5, 7, 9]) assert result.column(6) == pa.array([5, 7, 9]) def test_with_columns_str(df): df = df.with_columns( "a + b as c", "a + b as d", [ "a + b as e", "a + b as f", ], g="a + b", ) # execute and collect the first (and only) batch result = df.collect()[0] assert result.schema.field(0).name == "a" assert result.schema.field(1).name == "b" assert result.schema.field(2).name == "c" assert result.schema.field(3).name == "d" assert result.schema.field(4).name == "e" assert result.schema.field(5).name == "f" assert result.schema.field(6).name == "g" assert result.column(0) == pa.array([1, 2, 3]) assert result.column(1) == pa.array([4, 5, 6]) assert result.column(2) == pa.array([5, 7, 9]) assert result.column(3) == pa.array([5, 7, 9]) assert result.column(4) == pa.array([5, 7, 9]) assert result.column(5) == pa.array([5, 7, 9]) assert result.column(6) == pa.array([5, 7, 9]) def test_cast(df): df = df.cast({"a": pa.float16(), "b": pa.list_(pa.uint32())}) expected = pa.schema( [("a", pa.float16()), ("b", pa.list_(pa.uint32())), ("c", pa.int64())] ) assert df.schema() == expected def test_iter_batches(df): batches = [] for batch in df: batches.append(batch) # noqa: PERF402 # Delete DataFrame to ensure RecordBatches remain valid del df assert len(batches) == 1 batch = batches[0] assert isinstance(batch, RecordBatch) pa_batch = batch.to_pyarrow() assert pa_batch.column(0).to_pylist() == [1, 2, 3] assert pa_batch.column(1).to_pylist() == [4, 5, 6] assert pa_batch.column(2).to_pylist() == [8, 5, 8] def test_iter_returns_datafusion_recordbatch(df): for batch in df: assert isinstance(batch, RecordBatch) def test_execute_stream_basic(df): stream = df.execute_stream() batches = list(stream) assert len(batches) == 1 assert isinstance(batches[0], RecordBatch) pa_batch = batches[0].to_pyarrow() assert pa_batch.column(0).to_pylist() == [1, 2, 3] assert pa_batch.column(1).to_pylist() == [4, 5, 6] assert pa_batch.column(2).to_pylist() == [8, 5, 8] def test_with_column_renamed(df): df = df.with_column("c", column("a") + column("b")).with_column_renamed("c", "sum") result = df.collect()[0] assert result.schema.field(0).name == "a" assert result.schema.field(1).name == "b" assert result.schema.field(2).name == "sum" def test_unnest(nested_df): nested_df = nested_df.unnest_columns("a") # execute and collect the first (and only) batch result = nested_df.collect()[0] assert result.column(0) == pa.array([1, 2, 3, 4, 5, 6, None]) assert result.column(1) == pa.array([7, 8, 8, 9, 9, 9, 10]) def test_unnest_without_nulls(nested_df): nested_df = nested_df.unnest_columns("a", preserve_nulls=False) # execute and collect the first (and only) batch result = nested_df.collect()[0] assert result.column(0) == pa.array([1, 2, 3, 4, 5, 6]) assert result.column(1) == pa.array([7, 8, 8, 9, 9, 9]) def test_join(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df = ctx.create_dataframe([[batch]], "l") batch = pa.RecordBatch.from_arrays( [pa.array([1, 2]), pa.array([8, 10])], names=["a", "c"], ) df1 = ctx.create_dataframe([[batch]], "r") df2 = df.join(df1, on="a", how="inner") df2 = df2.sort(column("a")) table = pa.Table.from_batches(df2.collect()) expected = {"a": [1, 2], "c": [8, 10], "b": [4, 5]} assert table.to_pydict() == expected # Test the default behavior for dropping duplicate keys # Since we may have a duplicate column name and pa.Table() # hides the fact, instead we need to explicitly check the # resultant arrays. df2 = df.join( df1, left_on="a", right_on="a", how="inner", coalesce_duplicate_keys=True ) df2 = df2.sort(column("a")) result = df2.collect()[0] assert result.num_columns == 3 assert result.column(0) == pa.array([1, 2], pa.int64()) assert result.column(1) == pa.array([4, 5], pa.int64()) assert result.column(2) == pa.array([8, 10], pa.int64()) df2 = df.join( df1, left_on="a", right_on="a", how="inner", coalesce_duplicate_keys=False ) df2 = df2.sort(column("l.a")) result = df2.collect()[0] assert result.num_columns == 4 assert result.column(0) == pa.array([1, 2], pa.int64()) assert result.column(1) == pa.array([4, 5], pa.int64()) assert result.column(2) == pa.array([1, 2], pa.int64()) assert result.column(3) == pa.array([8, 10], pa.int64()) # Verify we don't make a breaking change to pre-43.0.0 # where users would pass join_keys as a positional argument df2 = df.join(df1, (["a"], ["a"]), how="inner") df2 = df2.sort(column("a")) table = pa.Table.from_batches(df2.collect()) expected = {"a": [1, 2], "c": [8, 10], "b": [4, 5]} assert table.to_pydict() == expected def test_join_invalid_params(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df = ctx.create_dataframe([[batch]], "l") batch = pa.RecordBatch.from_arrays( [pa.array([1, 2]), pa.array([8, 10])], names=["a", "c"], ) df1 = ctx.create_dataframe([[batch]], "r") with pytest.deprecated_call(): df2 = df.join(df1, join_keys=(["a"], ["a"]), how="inner") df2.show() df2 = df2.sort(column("a")) table = pa.Table.from_batches(df2.collect()) expected = {"a": [1, 2], "c": [8, 10], "b": [4, 5]} assert table.to_pydict() == expected with pytest.raises( ValueError, match=r"`left_on` or `right_on` should not provided with `on`" ): df2 = df.join(df1, on="a", how="inner", right_on="test") with pytest.raises( ValueError, match=r"`left_on` and `right_on` should both be provided." ): df2 = df.join(df1, left_on="a", how="inner") with pytest.raises( ValueError, match=r"either `on` or `left_on` and `right_on` should be provided." ): df2 = df.join(df1, how="inner") def test_join_on(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df = ctx.create_dataframe([[batch]], "l") batch = pa.RecordBatch.from_arrays( [pa.array([1, 2]), pa.array([-8, 10])], names=["a", "c"], ) df1 = ctx.create_dataframe([[batch]], "r") df2 = df.join_on(df1, column("l.a").__eq__(column("r.a")), how="inner") df2.show() df2 = df2.sort(column("l.a")) table = pa.Table.from_batches(df2.collect()) expected = {"a": [1, 2], "c": [-8, 10], "b": [4, 5]} assert table.to_pydict() == expected df3 = df.join_on( df1, column("l.a").__eq__(column("r.a")), column("l.a").__lt__(column("r.c")), how="inner", ) df3.show() df3 = df3.sort(column("l.a")) table = pa.Table.from_batches(df3.collect()) expected = {"a": [2], "c": [10], "b": [5]} assert table.to_pydict() == expected def test_join_full_with_drop_duplicate_keys(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 3, 5, 7, 9]), pa.array([True, True, True, True, True])], names=["log_time", "key_frame"], ) key_frame = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([2, 4, 6, 8, 10])], names=["log_time"], ) query_times = ctx.create_dataframe([[batch]]) merged = query_times.join( key_frame, left_on="log_time", right_on="log_time", how="full", coalesce_duplicate_keys=True, ) merged = merged.sort(column("log_time")) result = merged.collect()[0] assert result.num_columns == 2 assert result.column(0).to_pylist() == [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] def test_join_on_invalid_expr(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2]), pa.array([4, 5])], names=["a", "b"], ) df = ctx.create_dataframe([[batch]], "l") df1 = ctx.create_dataframe([[batch]], "r") with pytest.raises( TypeError, match=r"Use col\(\)/column\(\) or lit\(\)/literal\(\)" ): df.join_on(df1, "a") def test_aggregate_invalid_aggs(df): with pytest.raises( TypeError, match=r"Use col\(\)/column\(\) or lit\(\)/literal\(\)" ): df.aggregate([], "a") def test_distinct(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3, 1, 2, 3]), pa.array([4, 5, 6, 4, 5, 6])], names=["a", "b"], ) df_a = ctx.create_dataframe([[batch]]).distinct().sort(column("a")) batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df_b = ctx.create_dataframe([[batch]]).sort(column("a")) assert df_a.collect() == df_b.collect() data_test_window_functions = [ ( "row", f.row_number(order_by=[column("b"), column("a").sort(ascending=False)]), [4, 2, 3, 5, 7, 1, 6], ), ( "row_w_params", f.row_number( order_by=[column("b"), column("a")], partition_by=[column("c")], ), [2, 1, 3, 4, 2, 1, 3], ), ( "row_w_params_no_lists", f.row_number( order_by=column("b"), partition_by=column("c"), ), [2, 1, 3, 4, 2, 1, 3], ), ("rank", f.rank(order_by=[column("b")]), [3, 1, 3, 5, 6, 1, 6]), ( "rank_w_params", f.rank(order_by=[column("b"), column("a")], partition_by=[column("c")]), [2, 1, 3, 4, 2, 1, 3], ), ( "rank_w_params_no_lists", f.rank(order_by=column("a"), partition_by=column("c")), [1, 2, 3, 4, 1, 2, 3], ), ( "dense_rank", f.dense_rank(order_by=[column("b")]), [2, 1, 2, 3, 4, 1, 4], ), ( "dense_rank_w_params", f.dense_rank(order_by=[column("b"), column("a")], partition_by=[column("c")]), [2, 1, 3, 4, 2, 1, 3], ), ( "dense_rank_w_params_no_lists", f.dense_rank(order_by=column("a"), partition_by=column("c")), [1, 2, 3, 4, 1, 2, 3], ), ( "percent_rank", f.round(f.percent_rank(order_by=[column("b")]), literal(3)), [0.333, 0.0, 0.333, 0.667, 0.833, 0.0, 0.833], ), ( "percent_rank_w_params", f.round( f.percent_rank( order_by=[column("b"), column("a")], partition_by=[column("c")] ), literal(3), ), [0.333, 0.0, 0.667, 1.0, 0.5, 0.0, 1.0], ), ( "percent_rank_w_params_no_lists", f.round( f.percent_rank(order_by=column("a"), partition_by=column("c")), literal(3), ), [0.0, 0.333, 0.667, 1.0, 0.0, 0.5, 1.0], ), ( "cume_dist", f.round(f.cume_dist(order_by=[column("b")]), literal(3)), [0.571, 0.286, 0.571, 0.714, 1.0, 0.286, 1.0], ), ( "cume_dist_w_params", f.round( f.cume_dist( order_by=[column("b"), column("a")], partition_by=[column("c")] ), literal(3), ), [0.5, 0.25, 0.75, 1.0, 0.667, 0.333, 1.0], ), ( "cume_dist_w_params_no_lists", f.round( f.cume_dist(order_by=column("a"), partition_by=column("c")), literal(3), ), [0.25, 0.5, 0.75, 1.0, 0.333, 0.667, 1.0], ), ( "ntile", f.ntile(2, order_by=[column("b")]), [1, 1, 1, 2, 2, 1, 2], ), ( "ntile_w_params", f.ntile(2, order_by=[column("b"), column("a")], partition_by=[column("c")]), [1, 1, 2, 2, 1, 1, 2], ), ( "ntile_w_params_no_lists", f.ntile(2, order_by=column("b"), partition_by=column("c")), [1, 1, 2, 2, 1, 1, 2], ), ("lead", f.lead(column("b"), order_by=[column("b")]), [7, None, 8, 9, 9, 7, None]), ( "lead_w_params", f.lead( column("b"), shift_offset=2, default_value=-1, order_by=[column("b"), column("a")], partition_by=[column("c")], ), [8, 7, -1, -1, -1, 9, -1], ), ( "lead_w_params_no_lists", f.lead( column("b"), shift_offset=2, default_value=-1, order_by=column("b"), partition_by=column("c"), ), [8, 7, -1, -1, -1, 9, -1], ), ("lag", f.lag(column("b"), order_by=[column("b")]), [None, None, 7, 7, 8, None, 9]), ( "lag_w_params", f.lag( column("b"), shift_offset=2, default_value=-1, order_by=[column("b"), column("a")], partition_by=[column("c")], ), [-1, -1, None, 7, -1, -1, None], ), ( "lag_w_params_no_lists", f.lag( column("b"), shift_offset=2, default_value=-1, order_by=column("b"), partition_by=column("c"), ), [-1, -1, None, 7, -1, -1, None], ), ( "first_value", f.first_value(column("a")).over( Window(partition_by=[column("c")], order_by=[column("b")]) ), [1, 1, 1, 1, 5, 5, 5], ), ( "first_value_without_list_args", f.first_value(column("a")).over( Window(partition_by=column("c"), order_by=column("b")) ), [1, 1, 1, 1, 5, 5, 5], ), ( "first_value_order_by_string", f.first_value(column("a")).over( Window(partition_by=[column("c")], order_by="b") ), [1, 1, 1, 1, 5, 5, 5], ), ( "last_value", f.last_value(column("a")).over( Window( partition_by=[column("c")], order_by=[column("b")], window_frame=WindowFrame("rows", None, None), ) ), [3, 3, 3, 3, 6, 6, 6], ), ( "3rd_value", f.nth_value(column("b"), 3).over(Window(order_by=[column("a")])), [None, None, 7, 7, 7, 7, 7], ), ( "avg", f.round(f.avg(column("b")).over(Window(order_by=[column("a")])), literal(3)), [7.0, 7.0, 7.0, 7.333, 7.75, 7.75, 8.0], ), ] @pytest.mark.parametrize(("name", "expr", "result"), data_test_window_functions) def test_window_functions(partitioned_df, name, expr, result): df = partitioned_df.select( column("a"), column("b"), column("c"), f.alias(expr, name) ) df.sort(column("a")).show() table = pa.Table.from_batches(df.collect()) expected = { "a": [0, 1, 2, 3, 4, 5, 6], "b": [7, None, 7, 8, 9, None, 9], "c": ["A", "A", "A", "A", "B", "B", "B"], name: result, } assert table.sort_by("a").to_pydict() == expected @pytest.mark.parametrize("partition", ["c", df_col("c")]) def test_rank_partition_by_accepts_string(partitioned_df, partition): """Passing a string to partition_by should match using col().""" df = partitioned_df.select( f.rank(order_by=column("a"), partition_by=partition).alias("r") ) table = pa.Table.from_batches(df.sort(column("a")).collect()) assert table.column("r").to_pylist() == [1, 2, 3, 4, 1, 2, 3] @pytest.mark.parametrize("partition", ["c", df_col("c")]) def test_window_partition_by_accepts_string(partitioned_df, partition): """Window.partition_by accepts string identifiers.""" expr = f.first_value(column("a")).over( Window(partition_by=partition, order_by=column("b")) ) df = partitioned_df.select(expr.alias("fv")) table = pa.Table.from_batches(df.sort(column("a")).collect()) assert table.column("fv").to_pylist() == [1, 1, 1, 1, 5, 5, 5] @pytest.mark.parametrize( ("units", "start_bound", "end_bound"), [ (units, start_bound, end_bound) for units in ("rows", "range") for start_bound in (None, 0, 1) for end_bound in (None, 0, 1) ] + [ ("groups", 0, 0), ], ) def test_valid_window_frame(units, start_bound, end_bound): WindowFrame(units, start_bound, end_bound) @pytest.mark.parametrize( ("units", "start_bound", "end_bound"), [ ("invalid-units", 0, None), ("invalid-units", None, 0), ("invalid-units", None, None), ("groups", None, 0), ("groups", 0, None), ("groups", None, None), ], ) def test_invalid_window_frame(units, start_bound, end_bound): with pytest.raises(NotImplementedError, match=f"(?i){units}"): WindowFrame(units, start_bound, end_bound) def test_window_frame_defaults_match_postgres(partitioned_df): col_a = column("a") # When order is not set, the default frame should be unbounded preceding to # unbounded following. When order is set, the default frame is unbounded preceding # to current row. no_order = f.avg(col_a).over(Window()).alias("over_no_order") with_order = f.avg(col_a).over(Window(order_by=[col_a])).alias("over_with_order") df = partitioned_df.select(col_a, no_order, with_order) expected = { "a": [0, 1, 2, 3, 4, 5, 6], "over_no_order": [3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0], "over_with_order": [0.0, 0.5, 1.0, 1.5, 2.0, 2.5, 3.0], } assert df.sort(col_a).to_pydict() == expected def _build_last_value_df(df): return df.select( f.last_value(column("a")) .over( Window( partition_by=[column("c")], order_by=[column("b")], window_frame=WindowFrame("rows", None, None), ) ) .alias("expr"), f.last_value(column("a")) .over( Window( partition_by=[column("c")], order_by="b", window_frame=WindowFrame("rows", None, None), ) ) .alias("str"), ) def _build_nth_value_df(df): return df.select( f.nth_value(column("b"), 3).over(Window(order_by=[column("a")])).alias("expr"), f.nth_value(column("b"), 3).over(Window(order_by="a")).alias("str"), ) def _build_rank_df(df): return df.select( f.rank(order_by=[column("b")]).alias("expr"), f.rank(order_by="b").alias("str"), ) def _build_array_agg_df(df): return df.aggregate( [column("c")], [ f.array_agg(column("a"), order_by=[column("a")]).alias("expr"), f.array_agg(column("a"), order_by="a").alias("str"), ], ).sort(column("c")) @pytest.mark.parametrize( ("builder", "expected"), [ pytest.param(_build_last_value_df, [3, 3, 3, 3, 6, 6, 6], id="last_value"), pytest.param(_build_nth_value_df, [None, None, 7, 7, 7, 7, 7], id="nth_value"), pytest.param(_build_rank_df, [1, 1, 3, 3, 5, 6, 6], id="rank"), pytest.param(_build_array_agg_df, [[0, 1, 2, 3], [4, 5, 6]], id="array_agg"), ], ) def test_order_by_string_equivalence(partitioned_df, builder, expected): df = builder(partitioned_df) table = pa.Table.from_batches(df.collect()) assert table.column("expr").to_pylist() == expected assert table.column("expr").to_pylist() == table.column("str").to_pylist() def test_html_formatter_cell_dimension(df, clean_formatter_state): """Test configuring the HTML formatter with different options.""" # Configure with custom settings configure_formatter( max_width=500, max_height=200, enable_cell_expansion=False, ) html_output = df._repr_html_() # Verify our configuration was applied assert "max-height: 200px" in html_output assert "max-width: 500px" in html_output # With cell expansion disabled, we shouldn't see expandable-container elements assert "expandable-container" not in html_output def test_html_formatter_custom_style_provider(df, clean_formatter_state): """Test using custom style providers with the HTML formatter.""" # Configure with custom style provider configure_formatter(style_provider=CustomStyleProvider()) html_output = df._repr_html_() # Verify our custom styles were applied assert "background-color: #4285f4" in html_output assert "color: white" in html_output assert "background-color: #f5f5f5" in html_output def test_html_formatter_type_formatters(df, clean_formatter_state): """Test registering custom type formatters for specific data types.""" # Get current formatter and register custom formatters formatter = get_formatter() # Format integers with color based on value # Using int as the type for the formatter will work since we convert # Arrow scalar values to Python native types in _get_cell_value def format_int(value): return f' 2 else "blue"}">{value}' formatter.register_formatter(int, format_int) html_output = df._repr_html_() # Our test dataframe has values 1,2,3 so we should see: assert '1' in html_output def test_html_formatter_custom_cell_builder(df, clean_formatter_state): """Test using a custom cell builder function.""" # Create a custom cell builder with distinct styling for different value ranges def custom_cell_builder(value, row, col, table_id): try: num_value = int(value) if num_value > 5: # Values > 5 get green background with indicator return ( '' ) if num_value < 3: # Values < 3 get blue background with indicator return ( '' ) except (ValueError, TypeError): pass # Default styling for other cells (3, 4, 5) return f'' # Set our custom cell builder formatter = get_formatter() formatter.set_custom_cell_builder(custom_cell_builder) html_output = df._repr_html_() # Extract cells with specific styling using regex low_cells = re.findall( r'', html_output ) mid_cells = re.findall( r'', html_output ) high_cells = re.findall( r'', html_output ) # Sort the extracted values for consistent comparison low_cells = sorted(map(int, low_cells)) mid_cells = sorted(map(int, mid_cells)) high_cells = sorted(map(int, high_cells)) # Verify specific values have the correct styling applied assert low_cells == [1, 2] # Values < 3 assert mid_cells == [3, 4, 5, 5] # Values 3-5 assert high_cells == [6, 8, 8] # Values > 5 # Verify the exact content with styling appears in the output assert ( '' in html_output ) assert ( '' in html_output ) assert ( '' in html_output ) assert ( '' in html_output ) assert ( '' in html_output ) assert ( '' in html_output ) # Count occurrences to ensure all cells are properly styled assert html_output.count("-low") == 2 # Two low values (1, 2) assert html_output.count("-mid") == 4 # Four mid values (3, 4, 5, 5) assert html_output.count("-high") == 3 # Three high values (6, 8, 8) # Create a custom cell builder that changes background color based on value def custom_cell_builder(value, row, col, table_id): # Handle numeric values regardless of their exact type try: num_value = int(value) if num_value > 5: # Values > 5 get green background return f'' if num_value < 3: # Values < 3 get light blue background return f'' except (ValueError, TypeError): pass # Default styling for other cells return f'' # Set our custom cell builder formatter = get_formatter() formatter.set_custom_cell_builder(custom_cell_builder) html_output = df._repr_html_() # Verify our custom cell styling was applied assert "background-color: #d3e9f0" in html_output # For values 1,2 def test_html_formatter_custom_header_builder(df, clean_formatter_state): """Test using a custom header builder function.""" # Create a custom header builder with tooltips def custom_header_builder(field): tooltips = { "a": "Primary key column", "b": "Secondary values", "c": "Additional data", } tooltip = tooltips.get(field.name, "") return ( f'' ) # Set our custom header builder formatter = get_formatter() formatter.set_custom_header_builder(custom_header_builder) html_output = df._repr_html_() # Verify our custom headers were applied assert 'title="Primary key column"' in html_output assert 'title="Secondary values"' in html_output assert "background-color: #333; color: white" in html_output def test_html_formatter_complex_customization(df, clean_formatter_state): """Test combining multiple customization options together.""" # Create a dark mode style provider class DarkModeStyleProvider: def get_cell_style(self) -> str: return ( "background-color: #222; color: #eee; " "padding: 8px; border: 1px solid #444;" ) def get_header_style(self) -> str: return ( "background-color: #111; color: #fff; padding: 10px; " "border: 1px solid #333;" ) # Configure with dark mode style configure_formatter( max_cell_length=10, style_provider=DarkModeStyleProvider(), custom_css=""" .datafusion-table { font-family: monospace; border-collapse: collapse; } .datafusion-table tr:hover td { background-color: #444 !important; } """, ) # Add type formatters for special formatting - now working with native int values formatter = get_formatter() formatter.register_formatter( int, lambda n: f'{n}', ) html_output = df._repr_html_() # Verify our customizations were applied assert "background-color: #222" in html_output assert "background-color: #111" in html_output assert ".datafusion-table" in html_output assert "color: #5af" in html_output # Even numbers def test_html_formatter_memory(df, clean_formatter_state): """Test the memory and row control parameters in DataFrameHtmlFormatter.""" configure_formatter(max_memory_bytes=10, min_rows=1) html_output = df._repr_html_() # Count the number of table rows in the output tr_count = count_table_rows(html_output) # With a tiny memory limit of 10 bytes, the formatter should display # the minimum number of rows (1) plus a message about truncation assert tr_count == 2 # 1 for header row, 1 for data row assert "data truncated" in html_output.lower() configure_formatter(max_memory_bytes=10 * MB, min_rows=1) html_output = df._repr_html_() # With larger memory limit and min_rows=2, should display all rows tr_count = count_table_rows(html_output) # Table should have header row (1) + 3 data rows = 4 rows assert tr_count == 4 # No truncation message should appear assert "data truncated" not in html_output.lower() def test_html_formatter_memory_boundary_conditions(large_df, clean_formatter_state): """Test memory limit behavior at boundary conditions with large dataset. This test validates that the formatter correctly handles edge cases when the memory limit is reached with a large dataset (100,000 rows), ensuring that min_rows constraint is properly respected while respecting memory limits. Uses large_df to actually test memory limit behavior with realistic data sizes. """ # Get the raw size of the data to test boundary conditions # First, capture output with no limits # NOTE: max_rows=200000 is set well above the dataset size (100k rows) to ensure # we're testing memory limits, not row limits. Default max_rows=10 would # truncate before memory limit is reached. configure_formatter(max_memory_bytes=10 * MB, min_rows=1, max_rows=200000) unrestricted_output = large_df._repr_html_() unrestricted_rows = count_table_rows(unrestricted_output) # Test 1: Very small memory limit should still respect min_rows # With large dataset, this should definitely hit memory limit before min_rows configure_formatter(max_memory_bytes=10, min_rows=1) html_output = large_df._repr_html_() tr_count = count_table_rows(html_output) assert tr_count >= 2 # At least header + 1 data row (minimum) # Should show truncation since we limited memory so aggressively assert "data truncated" in html_output.lower() # Test 2: Memory limit at default size (2MB) should truncate the large dataset # Default max_rows would truncate at 10 rows, so we don't set it here to test # that memory limit is respected even with default row limit configure_formatter(max_memory_bytes=2 * MB, min_rows=1) html_output = large_df._repr_html_() tr_count = count_table_rows(html_output) assert tr_count >= 2 # At least header + min_rows # Should be truncated since full dataset is much larger than 2MB assert tr_count < unrestricted_rows # Test 3: Very large memory limit should show much more data # NOTE: max_rows=200000 is critical here - without it, default max_rows=10 # would limit output to 10 rows even though we have 100MB of memory available configure_formatter(max_memory_bytes=100 * MB, min_rows=1, max_rows=200000) html_output = large_df._repr_html_() tr_count = count_table_rows(html_output) # Should show significantly more rows, possibly all assert tr_count > 100 # Should show substantially more rows # Test 4: Min rows should override memory limit # With tiny memory and larger min_rows, min_rows should win configure_formatter(max_memory_bytes=10, min_rows=2) html_output = large_df._repr_html_() tr_count = count_table_rows(html_output) assert tr_count >= 3 # At least header + 2 data rows (min_rows) # Should show truncation message despite min_rows being satisfied assert "data truncated" in html_output.lower() # Test 5: With reasonable memory and min_rows settings # NOTE: max_rows=200000 ensures we test memory limit behavior, not row limit configure_formatter(max_memory_bytes=2 * MB, min_rows=10, max_rows=200000) html_output = large_df._repr_html_() tr_count = count_table_rows(html_output) assert tr_count >= 11 # header + at least 10 data rows (min_rows) # Should be truncated due to memory limit assert tr_count < unrestricted_rows def test_html_formatter_stream_early_termination( large_multi_batch_df, clean_formatter_state ): """Test that memory limits cause early stream termination with multi-batch data. This test specifically validates that the formatter stops collecting data when the memory limit is reached, rather than collecting all data and then truncating. The large_multi_batch_df fixture creates 10 record batches, allowing us to verify that not all batches are consumed when memory limit is hit. Key difference from test_html_formatter_memory_boundary_conditions: - Uses multi-batch DataFrame to verify stream termination behavior - Tests with memory limit exceeded by 2-3 batches but not 1 batch - Verifies partial data + truncation message + respects min_rows """ # Get baseline: how much data fits without memory limit configure_formatter(max_memory_bytes=100 * MB, min_rows=1, max_rows=200000) unrestricted_output = large_multi_batch_df._repr_html_() unrestricted_rows = count_table_rows(unrestricted_output) # Test 1: Memory limit exceeded by ~2 batches (each batch ~10k rows) # With 1 batch (~1-2MB), we should have space. With 2-3 batches, we exceed limit. # Set limit to ~3MB to ensure we collect ~1 batch before hitting limit configure_formatter(max_memory_bytes=3 * MB, min_rows=1, max_rows=200000) html_output = large_multi_batch_df._repr_html_() tr_count = count_table_rows(html_output) # Should show significant truncation (not all 100k rows) assert tr_count < unrestricted_rows, "Should be truncated by memory limit" assert tr_count >= 2, "Should respect min_rows" assert "data truncated" in html_output.lower(), "Should indicate truncation" # Test 2: Very tight memory limit should still respect min_rows # Even with tiny memory (10 bytes), should show at least min_rows configure_formatter(max_memory_bytes=10, min_rows=5, max_rows=200000) html_output = large_multi_batch_df._repr_html_() tr_count = count_table_rows(html_output) assert tr_count >= 6, "Should show header + at least min_rows (5)" assert "data truncated" in html_output.lower(), "Should indicate truncation" # Test 3: Memory limit should take precedence over max_rows in early termination # With max_rows=100 but small memory limit, should terminate early due to memory configure_formatter(max_memory_bytes=2 * MB, min_rows=1, max_rows=100) html_output = large_multi_batch_df._repr_html_() tr_count = count_table_rows(html_output) # Should be truncated by memory limit (showing more than max_rows would suggest # but less than unrestricted) assert tr_count >= 2, "Should respect min_rows" assert tr_count < unrestricted_rows, "Should be truncated" # Output should indicate why truncation occurred assert "data truncated" in html_output.lower() def test_html_formatter_max_rows(df, clean_formatter_state): configure_formatter(min_rows=2, max_rows=2) html_output = df._repr_html_() tr_count = count_table_rows(html_output) # Table should have header row (1) + 2 data rows = 3 rows assert tr_count == 3 configure_formatter(min_rows=2, max_rows=3) html_output = df._repr_html_() tr_count = count_table_rows(html_output) # Table should have header row (1) + 3 data rows = 4 rows assert tr_count == 4 def test_html_formatter_validation(): # Test validation for invalid parameters with pytest.raises(ValueError, match="max_cell_length must be a positive integer"): DataFrameHtmlFormatter(max_cell_length=0) with pytest.raises(ValueError, match="max_width must be a positive integer"): DataFrameHtmlFormatter(max_width=0) with pytest.raises(ValueError, match="max_height must be a positive integer"): DataFrameHtmlFormatter(max_height=0) with pytest.raises(ValueError, match="max_memory_bytes must be a positive integer"): DataFrameHtmlFormatter(max_memory_bytes=0) with pytest.raises(ValueError, match="max_memory_bytes must be a positive integer"): DataFrameHtmlFormatter(max_memory_bytes=-100) with pytest.raises(ValueError, match="min_rows must be a positive integer"): DataFrameHtmlFormatter(min_rows=0) with pytest.raises(ValueError, match="min_rows must be a positive integer"): DataFrameHtmlFormatter(min_rows=-5) with pytest.raises(ValueError, match="max_rows must be a positive integer"): DataFrameHtmlFormatter(max_rows=0) with pytest.raises(ValueError, match="max_rows must be a positive integer"): DataFrameHtmlFormatter(max_rows=-10) with pytest.raises( ValueError, match="min_rows must be less than or equal to max_rows" ): DataFrameHtmlFormatter(min_rows=5, max_rows=4) def test_repr_rows_backward_compatibility(clean_formatter_state): """Test that repr_rows parameter still works as deprecated alias.""" # Should work when not conflicting with max_rows with pytest.warns(DeprecationWarning, match="repr_rows parameter is deprecated"): formatter = DataFrameHtmlFormatter(repr_rows=15, min_rows=10) assert formatter.max_rows == 15 assert formatter.repr_rows == 15 # Should fail when conflicting with max_rows. The deprecation warning still # fires before the ValueError, so assert both. with ( pytest.raises(ValueError, match="Cannot specify both repr_rows and max_rows"), pytest.warns(DeprecationWarning, match="repr_rows parameter is deprecated"), ): DataFrameHtmlFormatter(repr_rows=5, max_rows=10) # Setting repr_rows via property should warn formatter2 = DataFrameHtmlFormatter() with pytest.warns(DeprecationWarning, match="repr_rows is deprecated"): formatter2.repr_rows = 7 assert formatter2.max_rows == 7 assert formatter2.repr_rows == 7 def test_configure_formatter(df, clean_formatter_state): """Test using custom style providers with the HTML formatter and configured parameters.""" # these are non-default values max_cell_length = 10 max_width = 500 max_height = 30 max_memory_bytes = 3 * MB min_rows = 2 max_rows = 2 enable_cell_expansion = False show_truncation_message = False use_shared_styles = False reset_formatter() formatter_default = get_formatter() assert formatter_default.max_cell_length != max_cell_length assert formatter_default.max_width != max_width assert formatter_default.max_height != max_height assert formatter_default.max_memory_bytes != max_memory_bytes assert formatter_default.min_rows != min_rows assert formatter_default.max_rows != max_rows assert formatter_default.enable_cell_expansion != enable_cell_expansion assert formatter_default.show_truncation_message != show_truncation_message assert formatter_default.use_shared_styles != use_shared_styles # Configure with custom style provider and additional parameters configure_formatter( max_cell_length=max_cell_length, max_width=max_width, max_height=max_height, max_memory_bytes=max_memory_bytes, min_rows=min_rows, max_rows=max_rows, enable_cell_expansion=enable_cell_expansion, show_truncation_message=show_truncation_message, use_shared_styles=use_shared_styles, ) formatter_custom = get_formatter() assert formatter_custom.max_cell_length == max_cell_length assert formatter_custom.max_width == max_width assert formatter_custom.max_height == max_height assert formatter_custom.max_memory_bytes == max_memory_bytes assert formatter_custom.min_rows == min_rows assert formatter_custom.max_rows == max_rows assert formatter_custom.enable_cell_expansion == enable_cell_expansion assert formatter_custom.show_truncation_message == show_truncation_message assert formatter_custom.use_shared_styles == use_shared_styles def test_configure_formatter_invalid_params(clean_formatter_state): """Test that configure_formatter rejects invalid parameters.""" with pytest.raises(ValueError, match="Invalid formatter parameters"): configure_formatter(invalid_param=123) # Test with multiple parameters, one valid and one invalid with pytest.raises(ValueError, match="Invalid formatter parameters"): configure_formatter(max_width=500, not_a_real_param="test") # Test with multiple invalid parameters with pytest.raises(ValueError, match="Invalid formatter parameters"): configure_formatter(fake_param1="test", fake_param2=456) def test_get_dataframe(tmp_path): ctx = SessionContext() path = tmp_path / "test.csv" table = pa.Table.from_arrays( [ [1, 2, 3, 4], ["a", "b", "c", "d"], [1.1, 2.2, 3.3, 4.4], ], names=["int", "str", "float"], ) write_csv(table, path) ctx.register_csv("csv", path) df = ctx.table("csv") assert isinstance(df, DataFrame) def test_struct_select(struct_df): df = struct_df.select( column("a")["c"] + column("b"), column("a")["c"] - column("b"), ) # execute and collect the first (and only) batch result = df.collect()[0] assert result.column(0) == pa.array([5, 7, 9]) assert result.column(1) == pa.array([-3, -3, -3]) def test_explain(df): df = df.select( column("a") + column("b"), column("a") - column("b"), ) df.explain() def test_logical_plan(aggregate_df): plan = aggregate_df.logical_plan() expected = "Projection: test.c1, sum(test.c2)" assert expected == plan.display() expected = ( "Projection: test.c1, sum(test.c2)\n" " Aggregate: groupBy=[[test.c1]], aggr=[[sum(test.c2)]]\n" " TableScan: test" ) assert expected == plan.display_indent() def test_optimized_logical_plan(aggregate_df): plan = aggregate_df.optimized_logical_plan() expected = "Aggregate: groupBy=[[test.c1]], aggr=[[sum(test.c2)]]" assert expected == plan.display() expected = ( "Aggregate: groupBy=[[test.c1]], aggr=[[sum(test.c2)]]\n" " TableScan: test projection=[c1, c2]" ) assert expected == plan.display_indent() def test_execution_plan(aggregate_df): plan = aggregate_df.execution_plan() expected = ( "AggregateExec: mode=FinalPartitioned, gby=[c1@0 as c1], aggr=[sum(test.c2)]\n" ) assert expected == plan.display() # Check the number of partitions is as expected. assert isinstance(plan.partition_count, int) expected = ( "ProjectionExec: expr=[c1@0 as c1, SUM(test.c2)@1 as SUM(test.c2)]\n" " Aggregate: groupBy=[[test.c1]], aggr=[[SUM(test.c2)]]\n" " TableScan: test projection=[c1, c2]" ) indent = plan.display_indent() # indent plan will be different for everyone due to absolute path # to filename, so we just check for some expected content assert "AggregateExec:" in indent assert "RepartitionExec:" in indent assert "DataSourceExec:" in indent assert "file_type=csv" in indent ctx = SessionContext() rows_returned = 0 for idx in range(plan.partition_count): stream = ctx.execute(plan, idx) try: batch = stream.next() assert batch is not None rows_returned += len(batch.to_pyarrow()[0]) except StopIteration: # This is one of the partitions with no values pass with pytest.raises(StopIteration): stream.next() assert rows_returned == 5 @pytest.mark.asyncio async def test_async_iteration_of_df(aggregate_df): rows_returned = 0 async for batch in aggregate_df: assert batch is not None rows_returned += len(batch.to_pyarrow()[0]) assert rows_returned == 5 def test_repartition(df): df.repartition(2) def test_repartition_by_hash(df): df.repartition_by_hash(column("a"), num=2) def test_repartition_by_hash_sql_expression(df): df.repartition_by_hash("a", num=2) def test_repartition_by_hash_mix(df): df.repartition_by_hash(column("a"), "b", num=2) def test_intersect(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df_a = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([3, 4, 5]), pa.array([6, 7, 8])], names=["a", "b"], ) df_b = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([3]), pa.array([6])], names=["a", "b"], ) df_c = ctx.create_dataframe([[batch]]).sort(column("a")) df_a_i_b = df_a.intersect(df_b).sort(column("a")) assert df_c.collect() == df_a_i_b.collect() def test_except_all(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df_a = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([3, 4, 5]), pa.array([6, 7, 8])], names=["a", "b"], ) df_b = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([1, 2]), pa.array([4, 5])], names=["a", "b"], ) df_c = ctx.create_dataframe([[batch]]).sort(column("a")) df_a_e_b = df_a.except_all(df_b).sort(column("a")) assert df_c.collect() == df_a_e_b.collect() def test_collect_partitioned(): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) assert [[batch]] == ctx.create_dataframe([[batch]]).collect_partitioned() def test_collect_column(ctx: SessionContext): batch_1 = pa.RecordBatch.from_pydict({"a": [1, 2, 3]}) batch_2 = pa.RecordBatch.from_pydict({"a": [4, 5, 6]}) batch_3 = pa.RecordBatch.from_pydict({"a": [7, 8, 9]}) ctx.register_record_batches("t", [[batch_1, batch_2], [batch_3]]) result = ctx.table("t").sort(column("a")).collect_column("a") expected = pa.array([1, 2, 3, 4, 5, 6, 7, 8, 9]) assert result == expected def test_union(ctx): batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df_a = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([3, 4, 5]), pa.array([6, 7, 8])], names=["a", "b"], ) df_b = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3, 3, 4, 5]), pa.array([4, 5, 6, 6, 7, 8])], names=["a", "b"], ) df_c = ctx.create_dataframe([[batch]]).sort(column("a")) df_a_u_b = df_a.union(df_b).sort(column("a")) assert df_c.collect() == df_a_u_b.collect() def test_union_distinct(ctx): batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3]), pa.array([4, 5, 6])], names=["a", "b"], ) df_a = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([3, 4, 5]), pa.array([6, 7, 8])], names=["a", "b"], ) df_b = ctx.create_dataframe([[batch]]) batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3, 4, 5]), pa.array([4, 5, 6, 7, 8])], names=["a", "b"], ) df_c = ctx.create_dataframe([[batch]]).sort(column("a")) df_a_u_b = df_a.union(df_b, distinct=True).sort(column("a")) assert df_c.collect() == df_a_u_b.collect() assert df_c.collect() == df_a_u_b.collect() def test_cache(df): assert df.cache().collect() == df.collect() def test_count(df): # Get number of rows assert df.count() == 3 def test_to_pandas(df): # Skip test if pandas is not installed pd = pytest.importorskip("pandas") # Convert datafusion dataframe to pandas dataframe pandas_df = df.to_pandas() assert isinstance(pandas_df, pd.DataFrame) assert pandas_df.shape == (3, 3) assert set(pandas_df.columns) == {"a", "b", "c"} def test_empty_to_pandas(df): # Skip test if pandas is not installed pd = pytest.importorskip("pandas") # Convert empty datafusion dataframe to pandas dataframe pandas_df = df.limit(0).to_pandas() assert isinstance(pandas_df, pd.DataFrame) assert pandas_df.shape == (0, 3) assert set(pandas_df.columns) == {"a", "b", "c"} def test_to_polars(df): # Skip test if polars is not installed pl = pytest.importorskip("polars") # Convert datafusion dataframe to polars dataframe polars_df = df.to_polars() assert isinstance(polars_df, pl.DataFrame) assert polars_df.shape == (3, 3) assert set(polars_df.columns) == {"a", "b", "c"} def test_empty_to_polars(df): # Skip test if polars is not installed pl = pytest.importorskip("polars") # Convert empty datafusion dataframe to polars dataframe polars_df = df.limit(0).to_polars() assert isinstance(polars_df, pl.DataFrame) assert polars_df.shape == (0, 3) assert set(polars_df.columns) == {"a", "b", "c"} def test_to_arrow_table(df): # Convert datafusion dataframe to pyarrow Table pyarrow_table = df.to_arrow_table() assert isinstance(pyarrow_table, pa.Table) assert pyarrow_table.shape == (3, 3) assert set(pyarrow_table.column_names) == {"a", "b", "c"} def test_parquet_non_null_column_to_pyarrow(ctx, tmp_path): path = tmp_path.joinpath("t.parquet") ctx.sql("create table t_(a int not null)").collect() ctx.sql("insert into t_ values (1), (2), (3)").collect() ctx.sql(f"copy (select * from t_) to '{path}'").collect() ctx.register_parquet("t", path) pyarrow_table = ctx.sql("select max(a) as m from t").to_arrow_table() assert pyarrow_table.to_pydict() == {"m": [3]} def test_parquet_empty_batch_to_pyarrow(ctx, tmp_path): path = tmp_path.joinpath("t.parquet") ctx.sql("create table t_(a int not null)").collect() ctx.sql("insert into t_ values (1), (2), (3)").collect() ctx.sql(f"copy (select * from t_) to '{path}'").collect() ctx.register_parquet("t", path) pyarrow_table = ctx.sql("select * from t limit 0").to_arrow_table() assert pyarrow_table.schema == pa.schema( [ pa.field("a", pa.int32(), nullable=False), ] ) def test_parquet_null_aggregation_to_pyarrow(ctx, tmp_path): path = tmp_path.joinpath("t.parquet") ctx.sql("create table t_(a int not null)").collect() ctx.sql("insert into t_ values (1), (2), (3)").collect() ctx.sql(f"copy (select * from t_) to '{path}'").collect() ctx.register_parquet("t", path) pyarrow_table = ctx.sql( "select max(a) as m from (select * from t where a < 0)" ).to_arrow_table() assert pyarrow_table.to_pydict() == {"m": [None]} assert pyarrow_table.schema == pa.schema( [ pa.field("m", pa.int32(), nullable=True), ] ) def test_execute_stream(df): stream = df.execute_stream() assert all(batch is not None for batch in stream) assert not list(stream) # after one iteration the generator must be exhausted @pytest.mark.asyncio async def test_execute_stream_async(df): stream = df.execute_stream() batches = [batch async for batch in stream] assert all(batch is not None for batch in batches) # After consuming all batches, the stream should be exhausted remaining_batches = [batch async for batch in stream] assert not remaining_batches @pytest.mark.parametrize("schema", [True, False]) def test_execute_stream_to_arrow_table(df, schema): stream = df.execute_stream() if schema: pyarrow_table = pa.Table.from_batches( (batch.to_pyarrow() for batch in stream), schema=df.schema() ) else: pyarrow_table = pa.Table.from_batches(batch.to_pyarrow() for batch in stream) assert isinstance(pyarrow_table, pa.Table) assert pyarrow_table.shape == (3, 3) assert set(pyarrow_table.column_names) == {"a", "b", "c"} @pytest.mark.asyncio @pytest.mark.parametrize("schema", [True, False]) async def test_execute_stream_to_arrow_table_async(df, schema): stream = df.execute_stream() if schema: pyarrow_table = pa.Table.from_batches( [batch.to_pyarrow() async for batch in stream], schema=df.schema() ) else: pyarrow_table = pa.Table.from_batches( [batch.to_pyarrow() async for batch in stream] ) assert isinstance(pyarrow_table, pa.Table) assert pyarrow_table.shape == (3, 3) assert set(pyarrow_table.column_names) == {"a", "b", "c"} def test_execute_stream_partitioned(df): streams = df.execute_stream_partitioned() assert all(batch is not None for stream in streams for batch in stream) assert all( not list(stream) for stream in streams ) # after one iteration all generators must be exhausted @pytest.mark.asyncio async def test_execute_stream_partitioned_async(df): streams = df.execute_stream_partitioned() for stream in streams: batches = [batch async for batch in stream] assert all(batch is not None for batch in batches) # Ensure the stream is exhausted after iteration remaining_batches = [batch async for batch in stream] assert not remaining_batches def test_empty_to_arrow_table(df): # Convert empty datafusion dataframe to pyarrow Table pyarrow_table = df.limit(0).to_arrow_table() assert isinstance(pyarrow_table, pa.Table) assert pyarrow_table.shape == (0, 3) assert set(pyarrow_table.column_names) == {"a", "b", "c"} def test_iter_batches_dataframe(fail_collect): ctx = SessionContext() batch1 = pa.record_batch([pa.array([1])], names=["a"]) batch2 = pa.record_batch([pa.array([2])], names=["a"]) df = ctx.create_dataframe([[batch1], [batch2]]) expected = [batch1, batch2] results = [b.to_pyarrow() for b in df] assert len(results) == len(expected) for exp in expected: assert any(got.equals(exp) for got in results) def test_arrow_c_stream_to_table_and_reader(fail_collect): ctx = SessionContext() # Create a DataFrame with two separate record batches batch1 = pa.record_batch([pa.array([1])], names=["a"]) batch2 = pa.record_batch([pa.array([2])], names=["a"]) df = ctx.create_dataframe([[batch1], [batch2]]) table = pa.Table.from_batches(batch.to_pyarrow() for batch in df) batches = table.to_batches() assert len(batches) == 2 expected = [batch1, batch2] for exp in expected: assert any(got.equals(exp) for got in batches) assert table.schema == df.schema() assert table.column("a").num_chunks == 2 reader = pa.RecordBatchReader.from_stream(df) assert isinstance(reader, pa.RecordBatchReader) reader_table = pa.Table.from_batches(reader) expected = pa.Table.from_batches([batch1, batch2]) assert reader_table.equals(expected) def test_arrow_c_stream_order(): ctx = SessionContext() batch1 = pa.record_batch([pa.array([1])], names=["a"]) batch2 = pa.record_batch([pa.array([2])], names=["a"]) df = ctx.create_dataframe([[batch1, batch2]]) table = pa.Table.from_batches(batch.to_pyarrow() for batch in df) expected = pa.Table.from_batches([batch1, batch2]) assert table.equals(expected) col = table.column("a") assert col.chunk(0)[0].as_py() == 1 assert col.chunk(1)[0].as_py() == 2 def test_arrow_c_stream_schema_selection(fail_collect): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [ pa.array([1, 2]), pa.array([3, 4]), pa.array([5, 6]), ], names=["a", "b", "c"], ) df = ctx.create_dataframe([[batch]]) requested_schema = pa.schema([("c", pa.int64()), ("a", pa.int64())]) c_schema = pa_cffi.ffi.new("struct ArrowSchema*") address = int(pa_cffi.ffi.cast("uintptr_t", c_schema)) requested_schema._export_to_c(address) capsule_new = ctypes.pythonapi.PyCapsule_New capsule_new.restype = ctypes.py_object capsule_new.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p] reader = pa.RecordBatchReader.from_stream(df, schema=requested_schema) assert reader.schema == requested_schema batches = list(reader) assert len(batches) == 1 expected_batch = pa.record_batch( [pa.array([5, 6]), pa.array([1, 2])], names=["c", "a"] ) assert batches[0].equals(expected_batch) def test_arrow_c_stream_schema_mismatch(fail_collect): ctx = SessionContext() batch = pa.RecordBatch.from_arrays( [pa.array([1, 2]), pa.array([3, 4])], names=["a", "b"] ) df = ctx.create_dataframe([[batch]]) bad_schema = pa.schema([("a", pa.string())]) c_schema = pa_cffi.ffi.new("struct ArrowSchema*") address = int(pa_cffi.ffi.cast("uintptr_t", c_schema)) bad_schema._export_to_c(address) capsule_new = ctypes.pythonapi.PyCapsule_New capsule_new.restype = ctypes.py_object capsule_new.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_void_p] bad_capsule = capsule_new(ctypes.c_void_p(address), b"arrow_schema", None) with pytest.raises(Exception, match="Fail to merge schema"): df.__arrow_c_stream__(bad_capsule) def test_to_pylist(df): # Convert datafusion dataframe to Python list pylist = df.to_pylist() assert isinstance(pylist, list) assert pylist == [ {"a": 1, "b": 4, "c": 8}, {"a": 2, "b": 5, "c": 5}, {"a": 3, "b": 6, "c": 8}, ] def test_to_pydict(df): # Convert datafusion dataframe to Python dictionary pydict = df.to_pydict() assert isinstance(pydict, dict) assert pydict == {"a": [1, 2, 3], "b": [4, 5, 6], "c": [8, 5, 8]} def test_describe(df): # Calculate statistics df = df.describe() # Collect the result result = df.to_pydict() assert result == { "describe": [ "count", "null_count", "mean", "std", "min", "max", "median", ], "a": [3.0, 0.0, 2.0, 1.0, 1.0, 3.0, 2.0], "b": [3.0, 0.0, 5.0, 1.0, 4.0, 6.0, 5.0], "c": [3.0, 0.0, 7.0, 1.7320508075688772, 5.0, 8.0, 8.0], } @pytest.mark.parametrize("path_to_str", [True, False]) def test_write_csv(ctx, df, tmp_path, path_to_str): path = str(tmp_path) if path_to_str else tmp_path df.write_csv(path, with_header=True) ctx.register_csv("csv", path) result = ctx.table("csv").to_pydict() expected = df.to_pydict() assert result == expected def generate_test_write_params() -> list[tuple]: # Overwrite and Replace are not implemented for many table writers insert_ops = [InsertOp.APPEND, None] sort_by_cases = [ (None, [1, 2, 3], "unsorted"), (column("c"), [2, 1, 3], "single_column_expr"), (column("a").sort(ascending=False), [3, 2, 1], "single_sort_expr"), ([column("c"), column("b")], [2, 1, 3], "list_col_expr"), ( [column("c").sort(ascending=False), column("b").sort(ascending=False)], [3, 1, 2], "list_sort_expr", ), ] formats = ["csv", "json", "parquet", "table"] return [ pytest.param( output_format, insert_op, sort_by, expected_a, id=f"{output_format}_{test_id}", ) for output_format, insert_op, ( sort_by, expected_a, test_id, ) in itertools.product(formats, insert_ops, sort_by_cases) ] @pytest.mark.parametrize( ("output_format", "insert_op", "sort_by", "expected_a"), generate_test_write_params(), ) def test_write_files_with_options( ctx, df, tmp_path, output_format, insert_op, sort_by, expected_a ) -> None: write_options = DataFrameWriteOptions(insert_operation=insert_op, sort_by=sort_by) if output_format == "csv": df.write_csv(tmp_path, with_header=True, write_options=write_options) ctx.register_csv("test_table", tmp_path) elif output_format == "json": df.write_json(tmp_path, write_options=write_options) ctx.register_json("test_table", tmp_path) elif output_format == "parquet": df.write_parquet(tmp_path, write_options=write_options) ctx.register_parquet("test_table", tmp_path) elif output_format == "table": batch = pa.RecordBatch.from_arrays([[], [], []], schema=df.schema()) ctx.register_record_batches("test_table", [[batch]]) ctx.table("test_table").show() df.write_table("test_table", write_options=write_options) result = ctx.table("test_table").to_pydict()["a"] ctx.table("test_table").show() assert result == expected_a @pytest.mark.parametrize("path_to_str", [True, False]) def test_write_json(ctx, df, tmp_path, path_to_str): path = str(tmp_path) if path_to_str else tmp_path df.write_json(path) ctx.register_json("json", path) result = ctx.table("json").to_pydict() expected = df.to_pydict() assert result == expected @pytest.mark.parametrize("path_to_str", [True, False]) def test_write_parquet(df, tmp_path, path_to_str): path = str(tmp_path) if path_to_str else tmp_path df.write_parquet(str(path)) result = pq.read_table(str(path)).to_pydict() expected = df.to_pydict() assert result == expected @pytest.mark.parametrize( ("compression", "compression_level"), [("gzip", 6), ("brotli", 7), ("zstd", 15)], ) def test_write_compressed_parquet(df, tmp_path, compression, compression_level): path = tmp_path df.write_parquet( str(path), compression=compression, compression_level=compression_level ) # test that the actual compression scheme is the one written for _root, _dirs, files in os.walk(path): for file in files: if file.endswith(".parquet"): metadata = pq.ParquetFile(tmp_path / file).metadata.to_dict() for row_group in metadata["row_groups"]: for columns in row_group["columns"]: assert columns["compression"].lower() == compression result = pq.read_table(str(path)).to_pydict() expected = df.to_pydict() assert result == expected @pytest.mark.parametrize( ("compression", "compression_level"), [("gzip", 12), ("brotli", 15), ("zstd", 23), ("wrong", 12)], ) def test_write_compressed_parquet_wrong_compression_level( df, tmp_path, compression, compression_level ): path = tmp_path with pytest.raises(ValueError): df.write_parquet( str(path), compression=compression, compression_level=compression_level, ) @pytest.mark.parametrize("compression", ["wrong"]) def test_write_compressed_parquet_invalid_compression(df, tmp_path, compression): path = tmp_path with pytest.raises(ValueError): df.write_parquet(str(path), compression=compression) # not testing lzo because it it not implemented yet # https://github.com/apache/arrow-rs/issues/6970 @pytest.mark.parametrize("compression", ["zstd", "brotli", "gzip"]) def test_write_compressed_parquet_default_compression_level(df, tmp_path, compression): # Test write_parquet with zstd, brotli, gzip default compression level, # ie don't specify compression level # should complete without error path = tmp_path df.write_parquet(str(path), compression=compression) def test_write_parquet_with_options_default_compression(df, tmp_path): """Test that the default compression is ZSTD.""" df.write_parquet(tmp_path) for file in tmp_path.rglob("*.parquet"): metadata = pq.ParquetFile(file).metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: assert col["compression"].lower() == "zstd" @pytest.mark.parametrize( "compression", ["gzip(6)", "brotli(7)", "zstd(15)", "snappy", "uncompressed"], ) def test_write_parquet_with_options_compression(df, tmp_path, compression): import re path = tmp_path df.write_parquet_with_options( str(path), ParquetWriterOptions(compression=compression) ) # test that the actual compression scheme is the one written for _root, _dirs, files in os.walk(path): for file in files: if file.endswith(".parquet"): metadata = pq.ParquetFile(tmp_path / file).metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: assert col["compression"].lower() == re.sub( r"\(\d+\)", "", compression ) result = pq.read_table(str(path)).to_pydict() expected = df.to_pydict() assert result == expected @pytest.mark.parametrize( "compression", ["gzip(12)", "brotli(15)", "zstd(23)"], ) def test_write_parquet_with_options_wrong_compression_level(df, tmp_path, compression): path = tmp_path with pytest.raises(Exception, match=r"valid compression range .*? exceeded."): df.write_parquet_with_options( str(path), ParquetWriterOptions(compression=compression) ) @pytest.mark.parametrize("compression", ["wrong", "wrong(12)"]) def test_write_parquet_with_options_invalid_compression(df, tmp_path, compression): path = tmp_path with pytest.raises(Exception, match="Unknown or unsupported parquet compression"): df.write_parquet_with_options( str(path), ParquetWriterOptions(compression=compression) ) @pytest.mark.parametrize( ("writer_version", "format_version"), [("1.0", "1.0"), ("2.0", "2.6"), (None, "1.0")], ) def test_write_parquet_with_options_writer_version( df, tmp_path, writer_version, format_version ): """Test the Parquet writer version. Note that writer_version=2.0 results in format_version=2.6""" if writer_version is None: df.write_parquet_with_options(tmp_path, ParquetWriterOptions()) else: df.write_parquet_with_options( tmp_path, ParquetWriterOptions(writer_version=writer_version) ) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() assert metadata["format_version"] == format_version @pytest.mark.parametrize("writer_version", ["1.2.3", "custom-version", "0"]) def test_write_parquet_with_options_wrong_writer_version(df, tmp_path, writer_version): """Test that invalid writer versions in Parquet throw an exception.""" with pytest.raises(Exception, match="Invalid parquet writer version"): df.write_parquet_with_options( tmp_path, ParquetWriterOptions(writer_version=writer_version) ) @pytest.mark.parametrize("dictionary_enabled", [True, False, None]) def test_write_parquet_with_options_dictionary_enabled( df, tmp_path, dictionary_enabled ): """Test enabling/disabling the dictionaries in Parquet.""" df.write_parquet_with_options( tmp_path, ParquetWriterOptions(dictionary_enabled=dictionary_enabled) ) # by default, the dictionary is enabled, so None results in True result = dictionary_enabled if dictionary_enabled is not None else True for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: assert col["has_dictionary_page"] == result @pytest.mark.parametrize( ("statistics_enabled", "has_statistics"), [("page", True), ("chunk", True), ("none", False), (None, True)], ) def test_write_parquet_with_options_statistics_enabled( df, tmp_path, statistics_enabled, has_statistics ): """Test configuring the statistics in Parquet. In pyarrow we can only check for column-level statistics, so "page" and "chunk" are tested in the same way.""" df.write_parquet_with_options( tmp_path, ParquetWriterOptions(statistics_enabled=statistics_enabled) ) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: if has_statistics: assert col["statistics"] is not None else: assert col["statistics"] is None @pytest.mark.parametrize("max_row_group_size", [1000, 5000, 10000, 100000]) def test_write_parquet_with_options_max_row_group_size( large_df, tmp_path, max_row_group_size ): """Test configuring the max number of rows per group in Parquet. These test cases guarantee that the number of rows for each row group is max_row_group_size, given the total number of rows is a multiple of max_row_group_size.""" path = f"{tmp_path}/t.parquet" large_df.write_parquet_with_options( path, ParquetWriterOptions(max_row_group_size=max_row_group_size) ) parquet = pq.ParquetFile(path) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: assert row_group["num_rows"] == max_row_group_size @pytest.mark.parametrize("created_by", ["datafusion", "datafusion-python", "custom"]) def test_write_parquet_with_options_created_by(df, tmp_path, created_by): """Test configuring the created by metadata in Parquet.""" df.write_parquet_with_options(tmp_path, ParquetWriterOptions(created_by=created_by)) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() assert metadata["created_by"] == created_by @pytest.mark.parametrize("statistics_truncate_length", [5, 25, 50]) def test_write_parquet_with_options_statistics_truncate_length( df, tmp_path, statistics_truncate_length ): """Test configuring the truncate limit in Parquet's row-group-level statistics.""" ctx = SessionContext() data = { "a": [ "a_the_quick_brown_fox_jumps_over_the_lazy_dog", "m_the_quick_brown_fox_jumps_over_the_lazy_dog", "z_the_quick_brown_fox_jumps_over_the_lazy_dog", ], "b": ["a_smaller", "m_smaller", "z_smaller"], } df = ctx.from_arrow(pa.record_batch(data)) df.write_parquet_with_options( tmp_path, ParquetWriterOptions(statistics_truncate_length=statistics_truncate_length), ) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: statistics = col["statistics"] assert len(statistics["min"]) <= statistics_truncate_length assert len(statistics["max"]) <= statistics_truncate_length def test_write_parquet_with_options_default_encoding(tmp_path): """Test that, by default, Parquet files are written with dictionary encoding. Note that dictionary encoding is not used for boolean values, so it is not tested here.""" ctx = SessionContext() data = { "a": [1, 2, 3], "b": ["1", "2", "3"], "c": [1.01, 2.02, 3.03], } df = ctx.from_arrow(pa.record_batch(data)) df.write_parquet_with_options(tmp_path, ParquetWriterOptions()) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: assert col["encodings"] == ("PLAIN", "RLE", "RLE_DICTIONARY") @pytest.mark.parametrize( ("encoding", "data_types", "result"), [ ("plain", ["int", "float", "str", "bool"], ("PLAIN", "RLE")), ("rle", ["bool"], ("RLE",)), ("delta_binary_packed", ["int"], ("RLE", "DELTA_BINARY_PACKED")), ("delta_length_byte_array", ["str"], ("RLE", "DELTA_LENGTH_BYTE_ARRAY")), ("delta_byte_array", ["str"], ("RLE", "DELTA_BYTE_ARRAY")), ("byte_stream_split", ["int", "float"], ("RLE", "BYTE_STREAM_SPLIT")), ], ) def test_write_parquet_with_options_encoding(tmp_path, encoding, data_types, result): """Test different encodings in Parquet in their respective support column types.""" ctx = SessionContext() data = {} for data_type in data_types: if data_type == "int": data["int"] = [1, 2, 3] elif data_type == "float": data["float"] = [1.01, 2.02, 3.03] elif data_type == "str": data["str"] = ["a", "b", "c"] elif data_type == "bool": data["bool"] = [True, False, True] df = ctx.from_arrow(pa.record_batch(data)) df.write_parquet_with_options( tmp_path, ParquetWriterOptions(encoding=encoding, dictionary_enabled=False) ) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: assert col["encodings"] == result @pytest.mark.parametrize("encoding", ["bit_packed"]) def test_write_parquet_with_options_unsupported_encoding(df, tmp_path, encoding): """Test that unsupported Parquet encodings do not work.""" # BaseException is used since this throws a Rust panic: https://github.com/PyO3/pyo3/issues/3519 with pytest.raises(BaseException, match=r"Encoding .*? is not supported"): df.write_parquet_with_options(tmp_path, ParquetWriterOptions(encoding=encoding)) @pytest.mark.parametrize("encoding", ["non_existent", "unknown", "plain123"]) def test_write_parquet_with_options_invalid_encoding(df, tmp_path, encoding): """Test that invalid Parquet encodings do not work.""" with pytest.raises(Exception, match="Unknown or unsupported parquet encoding"): df.write_parquet_with_options(tmp_path, ParquetWriterOptions(encoding=encoding)) @pytest.mark.parametrize("encoding", ["plain_dictionary", "rle_dictionary"]) def test_write_parquet_with_options_dictionary_encoding_fallback( df, tmp_path, encoding ): """Test that the dictionary encoding cannot be used as fallback in Parquet.""" # BaseException is used since this throws a Rust panic: https://github.com/PyO3/pyo3/issues/3519 with pytest.raises( BaseException, match="Dictionary encoding can not be used as fallback encoding" ): df.write_parquet_with_options(tmp_path, ParquetWriterOptions(encoding=encoding)) def test_write_parquet_with_options_bloom_filter(df, tmp_path): """Test Parquet files with and without (default) bloom filters. Since pyarrow does not expose any information about bloom filters, the easiest way to confirm that they are actually written is to compare the file size.""" path_no_bloom_filter = tmp_path / "1" path_bloom_filter = tmp_path / "2" df.write_parquet_with_options(path_no_bloom_filter, ParquetWriterOptions()) df.write_parquet_with_options( path_bloom_filter, ParquetWriterOptions(bloom_filter_on_write=True) ) size_no_bloom_filter = 0 for file in path_no_bloom_filter.rglob("*.parquet"): size_no_bloom_filter += Path(file).stat().st_size size_bloom_filter = 0 for file in path_bloom_filter.rglob("*.parquet"): size_bloom_filter += Path(file).stat().st_size assert size_no_bloom_filter < size_bloom_filter def test_write_parquet_with_options_column_options(df, tmp_path): """Test writing Parquet files with different options for each column, which replace the global configs (when provided).""" data = { "a": [1, 2, 3], "b": ["a", "b", "c"], "c": [False, True, False], "d": [1.01, 2.02, 3.03], "e": [4, 5, 6], } column_specific_options = { "a": ParquetColumnOptions(statistics_enabled="none"), "b": ParquetColumnOptions(encoding="plain", dictionary_enabled=False), "c": ParquetColumnOptions( compression="snappy", encoding="rle", dictionary_enabled=False ), "d": ParquetColumnOptions( compression="zstd(6)", encoding="byte_stream_split", dictionary_enabled=False, statistics_enabled="none", ), # column "e" will use the global configs } results = { "a": { "statistics": False, "compression": "brotli", "encodings": ("PLAIN", "RLE", "RLE_DICTIONARY"), }, "b": { "statistics": True, "compression": "brotli", "encodings": ("PLAIN", "RLE"), }, "c": { "statistics": True, "compression": "snappy", "encodings": ("RLE",), }, "d": { "statistics": False, "compression": "zstd", "encodings": ("RLE", "BYTE_STREAM_SPLIT"), }, "e": { "statistics": True, "compression": "brotli", "encodings": ("PLAIN", "RLE", "RLE_DICTIONARY"), }, } ctx = SessionContext() df = ctx.from_arrow(pa.record_batch(data)) df.write_parquet_with_options( tmp_path, ParquetWriterOptions( compression="brotli(8)", column_specific_options=column_specific_options ), ) for file in tmp_path.rglob("*.parquet"): parquet = pq.ParquetFile(file) metadata = parquet.metadata.to_dict() for row_group in metadata["row_groups"]: for col in row_group["columns"]: column_name = col["path_in_schema"] result = results[column_name] assert (col["statistics"] is not None) == result["statistics"] assert col["compression"].lower() == result["compression"].lower() assert col["encodings"] == result["encodings"] def test_write_parquet_options(df, tmp_path): options = ParquetWriterOptions(compression="gzip", compression_level=6) df.write_parquet(str(tmp_path), options) result = pq.read_table(str(tmp_path)).to_pydict() expected = df.to_pydict() assert result == expected def test_write_parquet_options_error(df, tmp_path): options = ParquetWriterOptions(compression="gzip", compression_level=6) with pytest.raises(ValueError): df.write_parquet(str(tmp_path), options, compression_level=1) def test_write_table(ctx, df): batch = pa.RecordBatch.from_arrays( [pa.array([1, 2, 3])], names=["a"], ) ctx.register_record_batches("t", [[batch]]) df = ctx.table("t").with_column("a", column("a") * literal(-1)) ctx.table("t").show() df.write_table("t") result = ctx.table("t").sort(column("a")).collect()[0][0].to_pylist() expected = [-3, -2, -1, 1, 2, 3] assert result == expected def test_dataframe_export(df) -> None: # Guarantees that we have the canonical implementation # reading our dataframe export table = pa.table(df) assert table.num_columns == 3 assert table.num_rows == 3 desired_schema = pa.schema([("a", pa.int64())]) # Verify we can request a schema table = pa.table(df, schema=desired_schema) assert table.num_columns == 1 assert table.num_rows == 3 # Expect a table of nulls if the schema don't overlap desired_schema = pa.schema([("g", pa.string())]) table = pa.table(df, schema=desired_schema) assert table.num_columns == 1 assert table.num_rows == 3 for i in range(3): assert table[0][i].as_py() is None # Expect an error when we cannot convert schema desired_schema = pa.schema([("a", pa.float32())]) failed_convert = False try: table = pa.table(df, schema=desired_schema) except Exception: failed_convert = True assert failed_convert # Expect an error when we have a not set non-nullable desired_schema = pa.schema([("g", pa.string(), False)]) failed_convert = False try: table = pa.table(df, schema=desired_schema) except Exception: failed_convert = True assert failed_convert def test_dataframe_transform(df): def add_string_col(df_internal) -> DataFrame: return df_internal.with_column("string_col", literal("string data")) def add_with_parameter(df_internal, value: Any) -> DataFrame: return df_internal.with_column("new_col", literal(value)) df = df.transform(add_string_col).transform(add_with_parameter, 3) result = df.to_pydict() assert result["a"] == [1, 2, 3] assert result["string_col"] == ["string data" for _i in range(3)] assert result["new_col"] == [3 for _i in range(3)] def test_dataframe_repr_html_structure(df, clean_formatter_state) -> None: """Test that DataFrame._repr_html_ produces expected HTML output structure.""" output = df._repr_html_() # Since we've added a fair bit of processing to the html output, lets just verify # the values we are expecting in the table exist. Use regex and ignore everything # between the and . We also don't want the closing > on the # td and th segments because that is where the formatting data is written. headers = ["a", "b", "c"] headers = [f"{v}" for v in headers] header_pattern = "(.*?)".join(headers) header_matches = re.findall(header_pattern, output, re.DOTALL) assert len(header_matches) == 1 # Update the pattern to handle values that may be wrapped in spans body_data = [[1, 4, 8], [2, 5, 5], [3, 6, 8]] body_lines = [ f"(?:]*?>)?{v}(?:)?" for inner in body_data for v in inner ] body_pattern = "(.*?)".join(body_lines) body_matches = re.findall(body_pattern, output, re.DOTALL) assert len(body_matches) == 1, "Expected pattern of values not found in HTML output" def test_dataframe_repr_html_values(df, clean_formatter_state): """Test that DataFrame._repr_html_ contains the expected data values.""" html = df._repr_html_() assert html is not None # Create a more flexible pattern that handles values being wrapped in spans # This pattern will match the sequence of values 1,4,8,2,5,5,3,6,8 regardless # of formatting pattern = re.compile( r"]*?>(?:]*?>)?1(?:)?.*?" r"]*?>(?:]*?>)?4(?:)?.*?" r"]*?>(?:]*?>)?8(?:)?.*?" r"]*?>(?:]*?>)?2(?:)?.*?" r"]*?>(?:]*?>)?5(?:)?.*?" r"]*?>(?:]*?>)?5(?:)?.*?" r"]*?>(?:]*?>)?3(?:)?.*?" r"]*?>(?:]*?>)?6(?:)?.*?" r"]*?>(?:]*?>)?8(?:)?", re.DOTALL, ) # Print debug info if the test fails matches = re.findall(pattern, html) if not matches: print(f"HTML output snippet: {html[:500]}...") # noqa: T201 assert len(matches) > 0, "Expected pattern of values not found in HTML output" def test_html_formatter_shared_styles(df, clean_formatter_state): """Test that shared styles work correctly across multiple tables.""" # First, ensure we're using shared styles configure_formatter(use_shared_styles=True) html_first = df._repr_html_() html_second = df._repr_html_() assert "
" f"{field.name}
" f"
" "" "" f"{formatted_value}" f"" f"
" f"		{formatted_value}
{value}-high{value}-low{value}-mid]*>(\d+)-low]*>(\d+)-mid]*>(\d+)-high1-low2-low3-mid4-mid6-high8-high{value}{value}{value}{field.name}