Skip to main content Skip to complementary content

Talend ESB runtime auto-scaling architecture on AWS platform

Main Talend Runtime Auto-Scaling process:

Procedure

  • Talend Administration Center has one or multiple pre-configured Runtime servers (from same Auto Scaling group) and web services applications (Talend Jobs or Routes) have been deployed from Nexus to each Runtime container. Existing Runtime(s) have AWS Elastic Load Balancer in front to distribute incoming traffic.
  • When one or more runtime(s) EC2 instances have passed resource capacity limitation pre-defined in AWS CloudWatch Events (E.g one of runtime EC2 instance’s CPU is running above 80% during the last 30 minutes - with increased Web Service requests during sales event).
  • AWS CloudWatch notifies Talend Auto Scaling Group (AWS Launch Configuration) to spin up one or more runtime server EC2 instances and use the same Load Balancer in front, so that increased incoming traffic could be redirected to new Runtime instances.
  • Once new Runtime Instance(s) are initialised by Auto Scaling group, it will use Talend MetaServlet as Linux service scripts to:
    1. Register itself in Talend Administration Center as a new Runtime server.
    2. Create a new task in Talend Administration Center/ESB Conductor and deploy the same service from Nexus repository to new Runtime container(s).
      Information noteNote: No human intervention is required during the Talend Runtime Auto-Scaling process.
  • Once incoming traffic is back to “normal” – E.g. All runtime server EC2 instance’s CPU are running under 20% for last hour - AWS CloudWatch Events will notify Auto Scaling group to stop/determine Runtime server instances and some Talend MetaServlet scripts can be used here:
    1. Undeploy the service from Runtime server.
    2. Remove the service from Talend Administration Center/ESB conductor.
    3. Deregister Talend Runtime in Talend Administration Center/Servers configuration page before shutdown.
      Information noteNote: For client requests traffic, all above steps will be keeping transparent during the entire process, since the AWS Load Balancer will be providing a unique entry service endpoint.

Did this page help you?

If you find any issues with this page or its content – a typo, a missing step, or a technical error – please let us know!

Leave your feedback here