BUG: new string dtype fails with >2 GB of data in a single column

[jorisvandenbossche]

(Patrick already opened PR #56220 with a change that addresses this, but posting it here anyway to have an example reproducer and some context for one of the reasons to switch to large string)

With the new String dtype enabled, and creating a Series with string data that holds more than 2GB of data:

import numpy as np
import string
data = ["".join(np.random.choice(list(string.ascii_letters), n)) for n in np.random.randint(10, 500, size=10_000)]

pd.options.future.infer_string = True
ser = pd.Series(data * 1000)
ser.memory_usage() / 1024**3
# 2.4261587597429752

then the take function fails:

In [28]: ser.take([0, 1, 2])
...
File ~/scipy/pandas/pandas/core/arrays/arrow/array.py:1199, in ArrowExtensionArray.take(self, indices, allow_fill, fill_value)
   1195         return result
   1196         # return type(self)(pc.fill_null(result, pa.scalar(fill_value)))
   1197     else:
   1198         # Nothing to fill
-> 1199         return type(self)(self._pa_array.take(indices))
   1200 else:  # allow_fill=False
   1201     # TODO(ARROW-9432): Treat negative indices as indices from the right.
   1202     if (indices_array < 0).any():
   1203         # Don't modify in-place

File ~/miniconda3/envs/dev311/lib/python3.11/site-packages/pyarrow/table.pxi:1029, in pyarrow.lib.ChunkedArray.take()

...

File ~/miniconda3/envs/dev311/lib/python3.11/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: offset overflow while concatenating arrays

This is because of a current limitation of the take compute function in Arrow for ChunkedArrays that cannot be combined into a single Array (apache/arrow#25822, apache/arrow#33049). And this issue specifically comes up with the Arrow string type, which uses int32 offsets and thus has a max of 2147483647 ((2 ** 32) / 2)) characters for all values in a single array, which is around 2G of data.
Arrow has a large_string type which uses int64 offsets instead, and essentially removes an upper limit (at least for pandas' use cases, it's in the petabytes).

And given that this happens in the take operation, this has a large consequence on various pandas operations (alignment, reindexing, joining, ...) that will fail with the current string dtype if you have a larger dataset.

xref #54792

[jorisvandenbossche]

Another occurrence of the same underlying issue is when a string columns gets created from some operation, for example starting from a column that fits in a single pyarrow StringArray, but becomes too big through the operation. The error you then see:

import numpy as np
import string
data = ["".join(np.random.choice(list(string.ascii_letters), 10)) for _ in range(1_000_000)]

pd.options.future.infer_string = True
ser = pd.Series(data * 100)  # <-- a bit more than 1GB, fits in a single chunk

>>> ser + "_some_suffix"
...
File ~/scipy/pandas/pandas/core/arrays/arrow/array.py:675, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
    673 sep = pa.scalar("", type=pa_type)
    674 if op is operator.add:
--> 675     result = pc.binary_join_element_wise(self._pa_array, other, sep)
    676 elif op is roperator.radd:
    677     result = pc.binary_join_element_wise(other, self._pa_array, sep)
...
ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2200000000

which fails because pyarrow currently hasn't any smart logic of splitting the result when needed (it's an element wise kernel, and so it preserves the input chunking).

[sqr00t]

Hiya, I notice that #56220 addressed the issue @jorisvandenbossche described, but I've encountered another occurance of ArrowCapacityError: array cannot contain more than 2147483646 bytes, therefore continuing in this thread. In short, caused by reading a parquet produced with pd.DataFrame.to_parquet(index=True)

I have a ~6.9GB .parquet file, while rather unorthodox, includes an index column produced by pd.DataFrame.to_parquet('file.parquet', index=True). The schema (col_name, dtype) is below, where description does contain some long strings from some webscraped data:

[
 ('id', Int64),
 ('description', String),
 ('__index_level_0__', Int64)
]

Reading this file.parquet with pd.read_parquet('file.parquet') causes the error.

1. I've instead tried using polars.read_parquet('file.parquet'), with default parameter use_pyarrow=False and all three columns loaded with no issue.
2. Switching to use_pyarrow=True surfaces the same issue.
3. Following 1.,

file_polars_df.select(['id', 'description']).write_parquet('file_fixed.parquet')

pd.read_parquet('file_fixed.parquet') # loads fine due to omitting `__index_level_0__`

4. However,

file_polars_df.select(['id', 'description']).to_pandas().to_parquet('file_with_index.parquet', index=True)

pd.read_parquet('file_with_index.parquet')

and I encounter the same ArrowCapacityError again.

Info on my system and env:
2021 MacBook Pro with M1 Pro 16GB RAM
Python: 3.8.18
pandas==2.0.3
pyarrow==14.0.1
polars==0.20.3

Edit: added stacktrace below

---------------------------------------------------------------------------
ArrowCapacityError                        Traceback (most recent call last)
Cell In[9], line 1
----> 1 test = pd.read_parquet('./file_with_index.parquet')

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pandas/io/parquet.py:509, in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, dtype_backend, **kwargs)
    506     use_nullable_dtypes = False
    507 check_dtype_backend(dtype_backend)
--> 509 return impl.read(
    510     path,
    511     columns=columns,
    512     storage_options=storage_options,
    513     use_nullable_dtypes=use_nullable_dtypes,
    514     dtype_backend=dtype_backend,
    515     **kwargs,
    516 )

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pandas/io/parquet.py:227, in PyArrowImpl.read(self, path, columns, use_nullable_dtypes, dtype_backend, storage_options, **kwargs)
    220 path_or_handle, handles, kwargs["filesystem"] = _get_path_or_handle(
    221     path,
    222     kwargs.pop("filesystem", None),
    223     storage_options=storage_options,
    224     mode="rb",
    225 )
    226 try:
--> 227     pa_table = self.api.parquet.read_table(
    228         path_or_handle, columns=columns, **kwargs
    229     )
    230     result = pa_table.to_pandas(**to_pandas_kwargs)
    232     if manager == "array":

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pyarrow/parquet/core.py:3003, in read_table(source, columns, use_threads, metadata, schema, use_pandas_metadata, read_dictionary, memory_map, buffer_size, partitioning, filesystem, filters, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties, thrift_string_size_limit, thrift_container_size_limit)
   2992         # TODO test that source is not a directory or a list
   2993         dataset = ParquetFile(
   2994             source, metadata=metadata, read_dictionary=read_dictionary,
   2995             memory_map=memory_map, buffer_size=buffer_size,
   (...)
   3000             thrift_container_size_limit=thrift_container_size_limit,
   3001         )
-> 3003     return dataset.read(columns=columns, use_threads=use_threads,
   3004                         use_pandas_metadata=use_pandas_metadata)
   3006 warnings.warn(
   3007     "Passing 'use_legacy_dataset=True' to get the legacy behaviour is "
   3008     "deprecated as of pyarrow 8.0.0, and the legacy implementation will "
   3009     "be removed in a future version.",
   3010     FutureWarning, stacklevel=2)
   3012 if ignore_prefixes is not None:

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pyarrow/parquet/core.py:2631, in _ParquetDatasetV2.read(self, columns, use_threads, use_pandas_metadata)
   2623         index_columns = [
   2624             col for col in _get_pandas_index_columns(metadata)
   2625             if not isinstance(col, dict)
   2626         ]
   2627         columns = (
   2628             list(columns) + list(set(index_columns) - set(columns))
   2629         )
-> 2631 table = self._dataset.to_table(
   2632     columns=columns, filter=self._filter_expression,
   2633     use_threads=use_threads
   2634 )
   2636 # if use_pandas_metadata, restore the pandas metadata (which gets
   2637 # lost if doing a specific `columns` selection in to_table)
   2638 if use_pandas_metadata:

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pyarrow/_dataset.pyx:556, in pyarrow._dataset.Dataset.to_table()

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pyarrow/_dataset.pyx:3713, in pyarrow._dataset.Scanner.to_table()

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pyarrow/error.pxi:154, in pyarrow.lib.pyarrow_internal_check_status()

File /opt/homebrew/Caskroom/miniconda/base/envs/some_env/lib/python3.8/site-packages/pyarrow/error.pxi:91, in pyarrow.lib.check_status()

ArrowCapacityError: array cannot contain more than 2147483646 bytes, have 2147483962

[phofl]

Can you provide a reproducer? We won't be able to help otherwise

It's possible that your DataFrame has arrow backed dtypes and that this uses pd.ArrowDtype(pa.string()) instead of large string, but this is outside of our control if your parquet file is saved like this

[jorisvandenbossche]

@sqr00t also, with which version of pandas and pyarrow are you reading this file? And did you enable certain pandas options? (because if you are not reading this with the latest pandas with the infer_string option set, it's probably not related to this issue, but an issue with pyarrow itself, given that the error comes from reading the parquet file itself with pyarrow before pandas is involved)

[sqr00t]

@phofl @jorisvandenbossche Thanks for both your replies and apologies for any confusion.

With pandas 2.1.4 (latest stable), and enable infer_string option with pd.set_option("future.infer_string", True) the file reads correctly. In my previous comment, I mentioned I was using pandas 2.0.3.

Appreciate the help!

[phofl]

We don't patch bugs in 2.0.x anymore, so the only thing you can do is upgrade