Cronjobs running on a single instance instead of all

[grigorisar]

My enquiry is that while my cloud system uses autoscaling to handle traffic, I want to trigger a cronjob that runs every 30 minutes that handles the integration with a 3rd party API. Now, my question is, does that cronjob abide by the autoscaling rules or does it run on its own instance.

As an example, let's say I have 4 instances open due to high traffic and the cronjob triggers. Would the cronjob run in all 4 instances of my application ? 

Basically what I want to avoid is having 4x the API calls for an integration that does not allow a large number of API calls.

[greb]

Hi @grigorisar,

Welcome to Google Cloud Community!

Your concern stems from the possibility of duplicate API calls when multiple instances execute the same cronjob. Here's an analysis and solution:

Possible Cause:
When autoscaling creates multiple instances of your application, each instance operates independently. If your cronjob logic is embedded within your application, it may execute on all active instances simultaneously, leading to duplicate API calls.

Recommendations:

1. Use Cloud Scheduler for Centralized Cron Management:
   Replace instance-level cronjobs with Cloud Scheduler. It triggers HTTP endpoints or Pub/Sub messages, ensuring only one execution regardless of the number of instances running.
2. Implement Distributed Locks:
   If you're managing custom environments (e.g., Kubernetes, Compute Engine), use distributed locking to designate one active instance for handling the cronjob.
3. Review and Optimize API Calls:
   Add logic to track and limit API calls to avoid exceeding the integration's allowed quota.

These steps may ensure your system remains efficient while avoiding redundant API calls. 

I hope the above information is helpful.