Skip to content

Commit

Permalink
Optimize experimental_python_types and add type-mapping tests
Browse files Browse the repository at this point in the history
Instead of checking the type for each row, check the type once for each
fetch() call and compute a list of lambdas which are to be applied to
the values from each row. A new RowMapperFactory class is created to
wrap this behavior.
The experimental_python_types flag is now processed in the TrinoQuery
class instead of the TrinoResult class.

Type mapping tests for each lambda which maps rows to Python types is
added.
  • Loading branch information
lpoulain authored and john-bodley committed Aug 31, 2022
1 parent f4487f5 commit 4545d7c
Show file tree
Hide file tree
Showing 9 changed files with 1,611 additions and 211 deletions.
23 changes: 5 additions & 18 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,21 +14,6 @@ concurrency:
cancel-in-progress: true

jobs:
checks:
runs-on: ubuntu-latest
steps:
- name: "Checkout the source code"
uses: actions/checkout@v2

- name: "Install Python"
uses: actions/setup-python@v2

- name: "Install pre-commit"
run: pip install pre-commit

- name: "Run pre-commit checks"
run: pre-commit run --all-files

build:
runs-on: ubuntu-latest
strategy:
Expand All @@ -49,6 +34,7 @@ jobs:
# Test with older Trino versions for backward compatibility
- { python: "3.10", trino: "351" } # first Trino version
env:
TOX_PARALLEL_NO_SPINNER: 1
TRINO_VERSION: "${{ matrix.trino }}"
steps:
- uses: actions/checkout@v2
Expand All @@ -59,7 +45,8 @@ jobs:
run: |
sudo apt-get update
sudo apt-get install libkrb5-dev
pip install .[tests]
- name: Run tests
sudo curl -sSL https://install.python-poetry.org | python3
poetry install
- name: Run tox
run: |
pytest -s tests/
poetry run tox --parallel
59 changes: 52 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -383,12 +383,12 @@ exits the *with* context and the queries succeed, otherwise

## Improved Python types

If you enable the flag `experimental_python_types`, the client will convert the results of the query to the
If you enable the flag `experimental_python_types`, the client will convert the results of the query to the
corresponding Python types. For example, if the query returns a `DECIMAL` column, the result will be a `Decimal` object.

Limitations of the Python types are described in the
[Python types documentation](https://docs.python.org/3/library/datatypes.html). These limitations will generate an
exception `trino.exceptions.DataError` if the query returns a value that cannot be converted to the corresponding Python
Limitations of the Python types are described in the
[Python types documentation](https://docs.python.org/3/library/datatypes.html). These limitations will generate an
exception `trino.exceptions.DataError` if the query returns a value that cannot be converted to the corresponding Python
type.

```python
Expand Down Expand Up @@ -418,6 +418,7 @@ assert cur.description[0][1] == "timestamp with time zone"

Start by forking the repository and then modify the code in your fork.

<<<<<<< HEAD
We recommend that you use Python3's `venv` for development:

```
Expand All @@ -430,6 +431,16 @@ With `-e` passed to `pip install` above pip can reference the code you are
modifying in the *virtual env*. That way, you do not need to run `pip install`
again to make your changes applied to the *virtual env*.

=======
Clone the repository and go inside the code directory.

Python dependencies are managed using [Poetry](https://python-poetry.org/) which helps to ensure the project is managed in a deterministic way. Poetry [creates a virtual environment](https://python-poetry.org/docs/managing-environments/) to aid with the process. Poetry should be installed via:

```
$ curl -sSL https://install.python-poetry.org | python3
```

>>>>>>> 1226d39 ([CI] Migrate to pyproject.toml and poetry for deterministic builds)
When the code is ready, submit a Pull Request.

### Code style
Expand All @@ -443,25 +454,37 @@ Most of them also apply to code in trino-python-client.

### Running tests

`trino-python-client` uses [pytest](https://pytest.org/) for its tests. To run
`trino-python-client` uses [tox](https://tox.wiki/en/latest/)—a tool for standardizing testing in Python—which leverages the [pytest](https://pytest.org/) testing framework. To run
only unit tests, type:

```
$ pytest tests/unit
$ poetry run tox -e <environment> -- tests/unit
```

Similarly to run only integration tests, type:

```
$ poetry run tox -e <environment> -- tests/integration
```

where `<environment>` denotes the Python environment (see the configuration in `tox.ini`).

Then you can pass options like `--pdb` or anything supported by `pytest --help`.

<<<<<<< HEAD
To run integration tests:

```
$ pytest tests/integration
```

=======
>>>>>>> 1226d39 ([CI] Migrate to pyproject.toml and poetry for deterministic builds)
They pull a Docker image and then run a container with a Trino server:
- the image is named `trinodb/trino:${TRINO_VERSION}`
- the container is named `trino-python-client-tests-{uuid4()[:7]}`

<<<<<<< HEAD
To run the tests with different versions of Python in managed *virtual envs*,
use `tox` (see the configuration in `tox.ini`):

Expand All @@ -470,13 +493,30 @@ $ tox
```

## Releasing
=======
### pre-commit

`trino-python-client` leverages [pre-commit](https://pre-commit.com/) to help identify simple issues before submission to code review. Checks include the validity of the `pyproject.toml` file, type checks via [Mypy](https://github.com/python/mypy), etc. To enable `pre-commit` run:

```
poetry run pre-commit install
```

which will run on every commit. You can also run it anytime using:

```
poetry run tox -e pre-commit
```

### Releasing
>>>>>>> 1226d39 ([CI] Migrate to pyproject.toml and poetry for deterministic builds)

- [Set up your development environment](#Getting-Started-With-Development).
- Check the local workspace is up to date and has no uncommitted changes
```bash
git fetch -a && git status
```
- Change version in `trino/__init__.py` to a new version, e.g. `0.123.0`.
- Change version in `trino/pyproject.toml` to a new version, e.g. `0.123.0`.
- Commit
```bash
git commit -a -m "Bump version to 0.123.0"
Expand All @@ -487,12 +527,17 @@ $ tox
```
- Create release package and upload it to PyPI
```bash
<<<<<<< HEAD
. .venv/bin/activate && \
pip install twine && \
rm -rf dist/ && \
./setup.py sdist bdist_wheel && \
twine upload dist/* && \
open https://pypi.org/project/trino/ && \
=======
poetry publish --build &&
open https://pypi.org/project/trino/ &&
>>>>>>> 1226d39 ([CI] Migrate to pyproject.toml and poetry for deterministic builds)
echo "Released!"
```
- Push the branch and the tag
Expand Down
Loading

0 comments on commit 4545d7c

Please sign in to comment.