WIP: [python-package] support Python 3.13

[jameslamb]

Add support for Python 3.13, now that it's available: https://docs.python.org/3/whatsnew/3.13.html

This also proposes a new standard process for Python version upgrades in this project. Proposing that when a new version is generally available, we do the following:

• bump every Python version in all CI up by 1 major version (e.g., all 3.10 -> 3.11)
• add any end-of-life versions to the test-old-versions CI job (as I've done here with Python 3.8)
  • based on https://devguide.python.org/versions/

This makes the updates easy to make and review, and reduces the risk of issues like #5969 and #6680 leading to maintenance work.

Notes for Reviewers

Just opening this to test. It's not ready for review... I expect more changes will be necessary, or at least that we'll need to wait for dependencies to update.

[jameslamb]

We'll need to wait for the rest of LightGBM's dependencies to get Python 3.13 packages on conda-forge.

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package mypy-1.11.1-py310h5b4e0ec_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ mypy >=1.11.1  is installable with the potential options
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python_abi 3.12.* *_cp312, which requires
│  │     └─ python 3.12.* *_cpython, which can be installed;
│  ├─ mypy [1.11.1|1.11.2] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2] would require
│  │  └─ pypy3.9 >=7.3.15 , which requires
│  │     └─ python 3.9.* *_73_pypy, which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  └─ mypy 1.12.0 would require
│     └─ python_abi 3.13.* *_cp313 with the potential options
│        ├─ python_abi 3.13 would require
│        │  └─ python 3.13.* *_cpython, which can be installed;
│        └─ python_abi 3.13 would require
│           └─ python 3.13.* *_cp313, which can be installed;
├─ pyarrow-core >=17.0  is installable with the potential options
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ __cuda >=11.8 , which is missing on the system;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python_abi 3.12.* *_cp312, which can be installed (as previously explained);
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  └─ pyarrow-core 17.0.0 would require
│     └─ python >=3.9,<3.10.0a0 , which can be installed;
└─ python 3.13 *_cp* is not installable because it conflicts with any installable versions previously reported.

(build link)

But hey, at least lightgbm itself already has Python 3.13 packages on conda-forge! (conda-forge/lightgbm-feedstock#63). That's thanks to lightgbm having such a small set of required dependencies... a good reminder that it's worthwhile to keep it that way 😁

We can check the state of LightGBM's dependencies here: https://conda-forge.org/status/migration/?name=python313

[jameslamb]

We can check the state of LightGBM's dependencies here: https://conda-forge.org/status/migration/?name=python313

pyarrow now has Python 3.13 packages (conda-forge/pyarrow-feedstock#139). I just merged latest master into this... let's see if all of LightGBM's recursive dependencies are now updated.

[jameslamb]

The conda-forge Python 3.13 migration is making progress, but looks like not all of lightgbm's dependencies have made it across yet. Looks like the next one we need is cffi.

PR to subscribe to: conda-forge/cffi-feedstock#53