Skip to main content

Workers

Nerve tracks all worker processes, their status, memory usage, and the jobs they’re processing.

Worker Status

workers = BrainzLab::Nerve.workers

workers.each do |worker|
  puts "#{worker.id}: #{worker.status}"
  puts "  Hostname: #{worker.hostname}"
  puts "  PID: #{worker.pid}"
  puts "  Queues: #{worker.queues.join(', ')}"
  puts "  Current job: #{worker.current_job}"
  puts "  Jobs processed: #{worker.processed}"
  puts "  Memory: #{worker.memory_mb}MB"
end

Worker Health

StatusDescription
activeProcessing jobs normally
idleRunning but no jobs to process
busyAll threads occupied
staleNo heartbeat (may be dead)
stoppedGracefully shut down

Aggregate Stats

stats = BrainzLab::Nerve.worker_stats

puts stats.total_workers      # => 5
puts stats.active_workers     # => 4
puts stats.idle_workers       # => 1
puts stats.total_threads      # => 25
puts stats.busy_threads       # => 18
puts stats.total_processed    # => 12345
puts stats.total_memory_mb    # => 2048

Worker Details

worker = BrainzLab::Nerve.worker("worker_abc123")

puts worker.id
puts worker.hostname
puts worker.pid
puts worker.started_at
puts worker.last_heartbeat
puts worker.concurrency      # Number of threads
puts worker.busy_threads     # Currently processing
puts worker.queues           # Assigned queues
puts worker.current_jobs     # Array of running jobs
puts worker.processed_count  # Jobs completed
puts worker.failed_count     # Jobs failed
puts worker.memory_mb        # Memory usage
puts worker.cpu_percent      # CPU usage

Worker History

View worker activity over time:
history = BrainzLab::Nerve.worker_history(period: :day)

history.each do |point|
  puts "#{point.timestamp}:"
  puts "  Workers: #{point.worker_count}"
  puts "  Busy: #{point.busy_percent}%"
  puts "  Throughput: #{point.jobs_per_minute}/min"
end

Scaling Recommendations

Get recommendations for worker scaling:
recommendations = BrainzLab::Nerve.scaling_recommendations

if recommendations.should_scale_up?
  puts "Recommendation: Add #{recommendations.workers_needed} workers"
  puts "Reason: #{recommendations.reason}"
  # => "Queue latency exceeds 30s for 'default'"
end

if recommendations.can_scale_down?
  puts "Recommendation: Remove #{recommendations.excess_workers} workers"
  puts "Reason: #{recommendations.reason}"
  # => "Workers averaging 10% utilization"
end

Worker Alerts

Set up alerts for worker issues:
# Alert when worker goes stale
BrainzLab::Nerve.alert_on_worker(
  condition: :stale,
  threshold: 60, # seconds without heartbeat
  channel: :slack
)

# Alert when memory exceeds limit
BrainzLab::Nerve.alert_on_worker(
  condition: :memory_exceeds,
  threshold: 512, # MB
  channel: :pagerduty
)

Quiet/Stop Workers

Gracefully manage workers:
# Signal worker to stop accepting new jobs
BrainzLab::Nerve.quiet_worker("worker_abc123")

# Signal all workers to quiet
BrainzLab::Nerve.quiet_all_workers

# Stop a worker after current jobs complete
BrainzLab::Nerve.stop_worker("worker_abc123")