Tagging AWS S3 objects in a file processing pipeline

This is just a quick tip how to keep your AWS file processing pipelines tied together in your logging and monitoring platform under one consistent trace. This is critical for investigation purposes. Also this is a real life scenario used often within AWS data processing operations pipelines. In this case, AWS Lambda A is a file generator ( a relational database data extraction tool ), Lambda B is processing additional file validation logic before this file gets send out. Boto3 calls in the Lambda functions are used to put and get the S3 object tags.

Diagram bez názvu

  1. Lambda function A generates a version 4 uuid used for the trace_id, starts logging under the trace_id and generates a csv file in a S3 bucket
  2. Lambda function A tags the csv file with a key “trace_id” and it’s value being the uuid
  3. Lambda function B gets the csv file
  4. Lambda function B reads the csv file tag with the trace_id and continues processing this file further while continuously logging under the same trace_id


I have one sidenote here: Step #3 and #4 could have had swapped order, depends on your use case though. Getting the tag of the object first will leave a minimal gap in the trace events, but might be more complex on the coding side of things.


AWS Glue job in a S3 event-driven scenario

I am working with PySpark under the hood of the AWS Glue service quite often recently and I spent some time trying to make such a Glue job s3-file-arrival-event-driven. I succeeded, the Glue job gets triggered on file arrival and I can guarantee that only the file that arrived gets processed, however the solution is not very straightforward. So this is the 10000 ft overview:

event_driven_glue (1) (1)


  1. File gets dropped to a s3 bucket “folder”, which is also set as a Glue table source in the Glue Data Catalog
  2. AWS Lambda gets triggered on this file arrival event, this lambda is doing this boto3 call besides some s3 key parsing, logging etc.
    def lambda_handler(event, context):
        # parsedjobname = .. parsed out from the "folder" name in the s3 file arrival event 
        # fullpath = .. parsed from the key in the s3 file arrival event
            glue_client.start_job_run(JobName=parsedjobname, Arguments={'--input_file_path': full_path})
            return 0
        except ClientError as e:
            logging.error("terminating - , %s", str(e))
            return 1
  3. The glue job corresponding to the “folder” name in the file arrival event gets triggered with this Job parameter set:glue_param
  4. The glue job loads into a Glue dynamic frame the content of the files from the AWS Glue data catalog like:
     datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "your_glue_db", table_name = "your_table_on_top_of_s3", transformation_ctx = "datasource0") 

    It also appends the filename to the dynamic frame, like this:

     from pyspark.sql.functions import input_file_name
    datasource1 = datasource0.toDF().withColumn("input_file_name", input_file_name()) 

    and at last, it converts the dynamic frame back to a dataframe like this:

     datasource2 = datasource0.fromDF(datasource1, glueContext, "datasource2") 
  5. In this step, we filter the dataframe to process further only the rows from the file related to the S3 file arrival event.
     datasource3 = Filter.apply(frame = datasource2, f = lambda x: x["input_file_name"] == args["input_file_path"]) 

    Let’s print out some metadata to the console for debugging purposes as well:

    print "input_file_path from AWS Lambda:" , args["input_file_path"]
    print "Filtered records count: ", datasource3.count()
  6. We can start to work with the filtered dataframe as we need in the Glue job now. You should consider to schedule some maintenance job or data retention policy on the file arrival bucket.

To guarantee that each file gets processed only once and never again ( that’s in case it would get dropped to the source bucket multiple times ) I would enhance the Lambda function with a logging write / lookup mechanism handling the filename ( or file content hash) in a DynamoDB logger table.

A few thoughts on AWS Batch with S3 event-driven usage scenarios

AWS Batch is a great service. This is what AWS says about it: AWS Batch enables developers, scientists, and engineers to easily and efficiently run hundreds of thousands of batch computing jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources (e.g., CPU or memory optimized instances) based on the volume and specific resource requirements of the batch jobs submitted. With AWS Batch, there is no need to install and manage batch computing software or server clusters that you use to run your jobs, allowing you to focus on analyzing results and solving problems. AWS Batch plans, schedules, and executes your batch computing workloads across the full range of AWS compute services and features, such as Amazon EC2 and Spot Instances.

What I want to write about in this blogpost is how to make the AWS Batch service work for you in a real-life S3 file arrival event-driven scenario. I use this approach for decoupling the metadata of the file that arrived to spin up a Batch data-processing job where the metadata from the file arrival event define the application logic and the  validations that are processed in the Batch job and when all succeeds, then the Batch job picks up the file itself for processing.

Let’s look at the 2 possible options I ‘ve worked with so far below:


Scenario #1 : A file arrives to a s3 bucket, CloudTrail logs capture the event and raise it to CloudWatch service, and this triggers AWS Batch job as it is a valid CloudWatch target. Use this scenario in case you don’t need to involve heavy logic in the arguments you pass to your Batch job. Typically you would use just basic metadata like the s3 key, s3 “file path” etc.

*Note: Don’t forget to have your CloudTrail log files repository in another bucket then the bucket you use for the file arrival event, otherwise the CloudTrail log files can easily keep triggering the Batch job 🙂

Scenario #2: A file arrives to a s3 bucket, Lambda function has this event set as an input, and this Lambda function triggers a AWS Batch job using the standard BOTO3 API library. Use this scenario when you need more logic before triggering the Batch job. Typically you might want to split the s3 file “file path”, or use the file size etc. and add some additional conditional logic for the arguments you provide to the Batch job.

Both of these solutions have some serious downside though. Solution #1 is weak in the way, that you are not able to add more complex conditional logic for the Batch job arguments. Solution #2 is weak in the way, that AWS Lambda Function has a 15 minute timeout , but the Batch job can run much longer, and therefore you never hear back from the Batch job execution in the context of the Lambda Function. So you’d have to have another Lambda function acting as a Batch job status poller. Ofcourse, you can follow up watching over the Batch job in CloudWatch logs or in the AWS Batch Dashboard, but in this case, you might want to try out the AWS Step functions. They allow you to add orchestration to your Lambda functions firing the Batch jobs. You can see more about AWS Step functions running Lambdas firing Batch jobs here .