How to Create and Upload an In Memory CSV File to Amazon S3 Bucket using Python

Introduction

In this tutorial we will create an in memory csv file and upload to Amazon S3 bucket using python package boto3. We will cover two scenarios here, 1). create an in memory file from nested lists (list of lists), 2). create in memory file from list of dictionaries, and then upload to Amazon S3.

1. Install dependencies

We need to install required dependencies in order to complete this tutorial. Install following package in your OS directly or first create virtual environment activate it and then install package in that virtual environment.

Run following command to create virtual environment (in windows)

virtualenv venv  # venv is name of virtual environment

Activate virtual environment:

.\venv\Scripts\activate.bat

Install following package(s) in virtual environment:

pip install boto3

2. Create In memory File

Create new file with extension .py and import following module:

import csv

import boto3

from io import StringIO

Create list of lists as following, first nested list will represent header of file, and other nested lists will represent data of file.

list_of_lists = [['name', 'age'], ['name 1', 25], ['name 2', 26], ['name 3', 27]]

If we have list of dictionaries, we can first convert that list of dictionaries to list of lists and then proceed with next steps. Use following code to convert list of dictionaries to list of lists.

# input list

list_of_dicts = [{'name': 'name 1', 'age': 25}, {'name': 'name 2', 'age': 26}, {'name': 'name 3', 'age': 27}]

# convert list of dictionary to list of lists

file_data = []

header = list(list_of_dicts[0].keys())

file_data = [[d[key] for key in header] for d in list_of_dicts]

file_data = [header] + file_data

At this point, we have converted list of dictionaries to list of lists, use following code to create in memory file and write data in it.

# create in memory file and write data in it.

file_to_save = StringIO()

csv.writer(file_to_save).writerows(file_data)

file_to_save = bytes(file_to_save.getvalue(), encoding='utf-8')

file_name_on_s3 = 'my_data.csv'

3. Save In Memory File to Amazon S3

We have created an in memory file, now use following code to save/upload that in memory file to Amazon S3.

# create boto3 client using your AWS access key id and secret access key

client = boto3.client('s3',

                      aws_access_key_id='your access key',

                      aws_secret_access_key='your secret key')

# save in memory file to S3

response = client.put_object(

Body=file_to_save,

Bucket='your bucket name',

Key=file_name_on_s3,

)

Conclusion

We have created an in memory file from list of lists and/or list of dictionaries, and uploaded to Amazon S3 bucket. Please let me know in comments if you have any better approach to implement this.

HAPPY CODING!!

See Also