How to Convert CSV to DynamoDB JSON Using Python

When working with AWS DynamoDB, you might find yourself needing to convert CSV files into DynamoDB JSON format. This process can be achieved using either an online converter or a Python script. In this article, we’ll walk you through both methods, helping you choose the best approach depending on your task size and complexity.

Using an Online CSV to DynamoDB JSON Converter

If you’re dealing with a small CSV file, using an online CSV to DynamoDB JSON converter can be a quick and easy solution. These tools are designed for convenience, allowing you to paste your CSV data and instantly get the DynamoDB JSON format.

For instance, a simple online tool might look like this:

You paste your CSV data into an input box.
Click the “Convert” button.
The tool then outputs the DynamoDB JSON format on the same page.

CSV to DynamoDB JSON Converter

Input CSV

Output DynamoDB JSON

This method is highly effective for smaller tasks, where the CSV file is not too large. However, if your file exceeds the maximum size supported by the online tool or if you need more customization, you might want to consider using Python.

Converting CSV to DynamoDB JSON Using Python

When you have a larger CSV file or need more control over the conversion process, using Python is the way to go. Here’s a step-by-step guide to convert CSV to DynamoDB JSON using Python and Visual Studio.

Step 1: Set Up Your Environment

Create a new folder on your device.

Open this folder using Visual Studio (File>New Window).

How to Convert CSV to DynamoDB JSON Using Python

Create a new Python file (ensure it has a .py extension).

Step 2: Python Script to Convert CSV to DynamoDB JSON

Here’s a sample Python script that you can use:

import pandas as pd
import json

def csv_to_dynamodb_json(csv_file, delimiter=',', quoting=0):
    """
    Convert a CSV file to DynamoDB JSON format.
   
    Parameters:
    - csv_file: Path to the CSV file.
    - delimiter: The delimiter used in the CSV file (default is comma).
    - quoting: Quoting option for pandas read_csv (default is 0, which means QUOTE_MINIMAL).
   
    Returns:
    - JSON formatted string representing the DynamoDB data.
    """
    try:
        # Read the CSV file
        df = pd.read_csv(csv_file, delimiter=delimiter, quoting=quoting, on_bad_lines='skip')

        # Convert DataFrame to DynamoDB JSON format
        dynamodb_json = df.to_dict(orient='records')

        # Convert to JSON string
        json_output = json.dumps(dynamodb_json, indent=4)

        return json_output

    except pd.errors.ParserError as e:
        print(f"Error parsing CSV file: {e}")
        return None
    except FileNotFoundError:
        print("CSV file not found.")
        return None
    except Exception as e:
        print(f"An unexpected error occurred: {e}")
        return None

if __name__ == "__main__":
    csv_file = r'C:\path\to\your\file.csv'  # Replace with the path to your CSV file
    delimiter = ','  # Replace with the delimiter if different
    quoting = 0  # Adjust quoting if necessary (0 for QUOTE_MINIMAL, 1 for QUOTE_ALL, 2 for QUOTE_NONNUMERIC, 3 for QUOTE_NONE)

    dynamodb_json = csv_to_dynamodb_json(csv_file, delimiter, quoting)
   
    if dynamodb_json:
        # Print the JSON or save it to a file
        print(dynamodb_json)
        with open('dynamodb_output.json', 'w') as json_file:
            json_file.write(dynamodb_json)

Step 3: Running the Script

Update the path in the code to your CSV file, ensuring it uses single quotes, like this: 'C:\path\to\your\file.csv', not double quotes.

Ensure you have Python installed, along with the necessary libraries (pandas and json).
Run the script in Visual Studio.

The output file in DynamoDB JSON format will be saved in your chosen directory.

This method gives you more flexibility, especially when working with large files that might exceed the maximum size limits of online tools. It also allows for customization of the CSV reading process, such as handling different delimiters or quoting options.

Conclusion

Whether you choose to use an online tool or a Python script depends on the size and complexity of your task. For quick and simple conversions, an online CSV to DynamoDB JSON converter is highly convenient. However, for larger or more complex tasks, using Python offers greater control and flexibility.

Both methods are invaluable when working with AWS DynamoDB, ensuring your data is correctly formatted and ready for database operations. The next time you need to convert CSV to DynamoDB JSON, you'll know exactly which tool to reach for, whether it's a quick online converter or a powerful Python script.