Setting Up Automated Model Training Workflows with AWS S3 | by Khuyen Tran | Mar, 2024

The Open-Supply Method for Workflow Automation

Take into account you’re an e-commerce platform aiming to reinforce advice personalization. Your knowledge resides in S3.

To refine suggestions, you intend to retrain advice fashions utilizing recent buyer interplay knowledge every time a brand new file is added to S3. However how precisely do you strategy this job?

Two widespread options to this drawback are:

1. AWS Lambda: A serverless compute service by AWS, permitting code execution in response to occasions with out managing servers.
2. Open-source orchestrators: Instruments automating, scheduling, and monitoring workflows and duties, normally self-hosted.

Utilizing an open-source orchestrator provides benefits over AWS Lambda:

• Value-Effectiveness: Operating lengthy duties on AWS Lambda may be expensive. Open-source orchestrators allow you to use your infrastructure, doubtlessly saving prices.
• Quicker Iteration: Growing and testing workflows domestically hastens the method, making it simpler to debug and refine.
• Setting Management: Full management over the execution setting means that you can customise your growth instruments and IDEs to match your preferences.

When you may remedy this drawback in Apache Airflow, it could require advanced infrastructure and deployment setup. Thus, we’ll use Kestra, which provides an intuitive UI and may be launched in a single Docker command.

Be at liberty to play and fork the supply code of this text right here:

This workflow consists of two foremost elements: Python scripts and orchestration.