AWS

31.08.2026

Amazon Web Service (AWS) for Data Engineering

Throughout the different posts, we will work with services such as Amazon S3, IAM, AWS Glue, Amazon Athena, and other key components of the AWS ecosystem. Each article will include practical examples, code snippets, and clear explanations, allowing you to reproduce the exercises in your own AWS account.

The goal is that, by following this step-by-step approach, you will be able to design, implement, and maintain scalable and secure data solutions on AWS, both for personal projects and enterprise environments.

At the end of this post, you will find instructions for releasing the resources used throughout the exercises, in order to avoid additional charges to your AWS account.

In this example, we use a free dataset called Brazilian E-Commerce Public Dataset by Olist, wich is available in Kaggle.com.

www.kaggle.com/datasets/olistbr/brazilian-ecommerce

Amazon S3 as Data Lake

1. After signing up for Amazon Web service, type 'S3' into the search bar and select S3 service and click 'create a bucket'

2. The bucket name is proyecto-olis-datos-2026, but you can use any other name you prefer. Keep all the default setting and click 'Create bucket'

Once the bucket has been created, the following table should appear.

3. Creating Folders in the proyecto-olis-datos-2026 Bucket Using CloudShell

Script Bash AWS S3

AWS CLI bash — create-s3-folders.sh

# Definir la variable del bucket

BUCKET_NAME="proyecto-olist-datos-2026"

# Crear la carpeta procesada

aws s3api put-object --bucket $BUCKET_NAME --key processed/

# Crear la carpeta raw y sus subcarpetas

aws s3api put-object --bucket $BUCKET_NAME --key raw/

aws s3api put-object --bucket $BUCKET_NAME --key raw/orders/

aws s3api put-object --bucket $BUCKET_NAME --key raw/customers/

💡 Define el nombre del bucket S3 y crea la estructura jerárquica de carpetas necesaria para almacenar las zonas de datos crudos y procesados.

The command has created two folder called raw and processed. Inside the raw folder , It has creates two subfolder called orders and customers.

4. Verifying folder creation inside proyecto-olist-datos-2026 bucket

AWS S3 List Output

AWS CLI cloudshell — s3-ls.sh

~ $ aws s3 ls s3://$BUCKET_NAME/ --recursive

2026-08-30 04:32:47 0 processed/

2026-08-30 04:33:11 0 raw/

2026-08-30 04:33:45 0 raw/customers/

2026-08-30 04:33:32 0 raw/orders/

5.Upload the files olist_orders_dataset.csv to raw/orders/ and olist_customers_dataset.csv to raw/customers/

Creating IAM security roles for AWS Glue

1.Type 'IAM' into the search bar and select IAM service and select roles option. Select trusted entity and select Glue as service or use case and Glue

2. Select the permission policies. In this case, we will select:

AWSGlueServiceRole
AmazonS3FullAccess

3.Name and create the role: GlueServiceRoleProject

Once the role has been created, it should appear in the list of roles, as shown in the following image:

4. Creating and running an ETL job in AWS Glue

Create the pyspark script to combine the dataset uploaded to /raw/orders/ and /raw/customers/ folders within S3 bucket named proyecto-olist-datos-2026

Proceso ETL con AWS Glue y PySpark

El siguiente script utiliza AWS Glue y PySpark para leer dos datasets almacenados en Amazon S3, realizar una unificación mediante un JOIN y almacenar el resultado procesado nuevamente en S3 en formato Parquet.

Script PySpark

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from pyspark.sql.functions import col

# ---------------------------------------------------------
# 1. Inicializar Spark y Glue
# ---------------------------------------------------------

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session

# ---------------------------------------------------------
# 2. Nombre del bucket S3
# ---------------------------------------------------------

# Reemplaza por el nombre exacto de tu bucket S3
BUCKET_NAME = "proyecto-olist-datos-2026"

# ---------------------------------------------------------
# 3. Leer los archivos CSV desde S3
# ---------------------------------------------------------

df_orders = (
    spark.read
    .option("header", "true")
    .option("inferSchema", "true")
    .csv(f"s3://{BUCKET_NAME}/raw/orders/")
)

df_customers = (
    spark.read
    .option("header", "true")
    .option("inferSchema", "true")
    .csv(f"s3://{BUCKET_NAME}/raw/customers/")
)

# ---------------------------------------------------------
# 4. Realizar la unificación (JOIN) por customer_id
# ---------------------------------------------------------

df_joined = df_orders.join(
    df_customers,
    on="customer_id",
    how="inner"
)

# ---------------------------------------------------------
# 5. Seleccionar columnas relevantes
# ---------------------------------------------------------

df_final = df_joined.select(
    col("order_id"),
    col("customer_id"),
    col("customer_unique_id"),
    col("order_status"),
    col("order_purchase_timestamp"),
    col("customer_zip_code_prefix"),
    col("customer_city"),
    col("customer_state")
)

# ---------------------------------------------------------
# 6. Mostrar resultado
# ---------------------------------------------------------

df_final.show(10, truncate=False)

# ---------------------------------------------------------
# 7. Guardar dataset procesado en S3
# ---------------------------------------------------------

output_path = f"s3://{BUCKET_NAME}/processed/orders_customers/"

df_final.write \
    .mode("overwrite") \
    .parquet(output_path)

print("Proceso ETL finalizado correctamente.")
print(f"Datos guardados en: {output_path}")

Descripción del proceso

Inicialización: Se configura el entorno de Spark y AWS Glue mediante SparkContext y GlueContext.
Lectura desde S3: Se leen los archivos CSV almacenados en las carpetas /raw/orders/ y /raw/customers/.
Transformación: Los datasets se unen mediante un INNER JOIN utilizando la columna customer_id.
Selección: Se seleccionan únicamente las columnas necesarias para el dataset final.
Almacenamiento: El resultado se guarda en Amazon S3 en formato Parquet, dentro de la carpeta /processed/orders_customers/.

Selecting the appropriate region is essential. In this example, the US East (N. Virginia) region will be used.