Add support for jitter so that we can ensure evenly distributed tasks don't cause all workers to restart at same time

[adiberk]

Summary

This PR adds jitter support to the max_async_tasks parameter to prevent synchronized worker restarts. When multiple workers are configured with the same max_async_tasks limit and receive evenly distributed tasks, they tend to hit their limits simultaneously, causing cascading restarts that can impact system stability.

Changes

• New max_async_tasks_jitter parameter: Added across the receiver, broker, and CLI interfaces to configure randomized jitter
• Randomized semaphore limits: The receiver now applies a random value (0 to jitter value) to the base max_async_tasks limit when creating the semaphore
• CLI argument: Added --max-async-tasks-jitter flag to worker command for runtime configuration

Technical Details

The implementation distributes worker restart times by randomizing each worker's actual task limit. For example, with max_async_tasks=100 and max_async_tasks_jitter=10, workers will have limits ranging from 100 to 110. This prevents the thundering herd problem where synchronized restarts cause all workers to compete for the same resources simultaneously, leading to more graceful degradation under load.

[adiberk]

@s3rius I would be curious as to whether you think this is valuable or not?

I hit an issue where in ensuring even distribution, my workers across 3 replicas would end up shutting down around the same time given the extremely high task usage

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.41%. Comparing base (eaed387) to head (a5c7546).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #570      +/-   ##
==========================================
+ Coverage   79.37%   79.41%   +0.03%     
==========================================
  Files          69       69              
  Lines        2473     2477       +4     
==========================================
+ Hits         1963     1967       +4     
  Misses        510      510              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.