boto3 client with smart_open memory leak

[jia-zhengwei]

Describe the bug

When use boto3 to read remote oss parquet file get memory leak.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

No memory leak

Current Behavior

As the method be called many times, the memory leak happend.

Reproduction Steps

Run this script many times

self.client = boto3.client(
    "s3",
    aws_access_key_id=access_key_id,
    aws_secret_access_key=secret_access_key,
    endpoint_url=endpoint,
    config=Config(
        s3={"addressing_style": "virtual", "signature_version": "s3v4"}
        ),
)
with smart_open.open(
    f"s3://{bucket_name}/{file_path}",
    mode="rb",
    transport_params={"client": self.client},
) as f:
    parquet_file = pq.ParquetFile(f)
    ...

self.client.close()

Possible Solution

I guess may related to smart_open object not be closed

Additional Information/Context

no

SDK version used

boto3==1.34.54 botocore==1.34.54

Environment details (OS name and version, etc.)

centos

[tim-finnigan]

Thanks for reaching out. Can you please share your debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script to help with further investigation? And can you share any more details regarding the memory leak? If you're able to provide a memory profiling analysis that could be helpful as well. Otherwise if the issue here is with smart_open then you may want to consider opening an issue in that repository.

[jia-zhengwei]

After added log(boto3.set_stream_logger("boto3", level=logging.DEBUG)), it shows nothing new log.

It's not smart_open bug.

I have a class which will init new instance of boto3 client, and each call will raise the memory up.

def print_memory_used():
    import threading

    import psutil

    def monitor_memory():
        pid = os.getpid()
        process = psutil.Process(pid)

        while True:
            memory_info = process.memory_info()
            memory_usage_mb = memory_info.rss / (1024**2)  # MB
            logger.info(
                f"[Memory Monitor] Process ID: {pid}, Memory Usage: {memory_usage_mb:.2f} MB"  # noqa 
            )
            time.sleep(1)

    memory_thread = threading.Thread(target=monitor_memory, daemon=True)
    memory_thread.start()