JOB QUEUE

web is more than request and response

Abhinav Lal
WHO AM I
 CTO at Naabo
 TurboDoc.in
 Cooking PHP for 4 years
Job Queue

In computer science, message
queues and mailboxes are software-
engineering components used for interprocess
communication, or for inter-thread communication within
the same process.
http://en.wikipedia.org/wiki/Message_queue

Any long running work which does not require user

feedback should be done in background.
Order processing without job
queue
New Order

Store data in database

Generate receipt

Provision new settings for user

Email user with order details

Send thank you message back to browser

Order processing with job queue

New Order
Job Queue

Store data in database

Generate Receipt
Add receipt generation job

Add Provision user job Provision user

Add email user job

Email user

Send thank you message back to browser

Jobs to run in background

 Sending email
 Fetching RSS feed periodically
 Interacting with other application API
 Complex report generation
 Log analysis
 cleaning file system
o periodically or after certain action.
But I can use cron for that

 Asynchronous
 Low latency
 Reliable
 Scalable
Options out there

Active MQ
Memcache
Beanstalkd Gearman
Queue

Amazon Rabbit OpenAM

Dropr Zero MQ
SQS MQ Q
Options out there

Active MQ
Memcache
Beanstalkd Gearman
Queue

Amazon Rabbit OpenAM

Dropr Zero MQ
SQS MQ Q
Gearman = Manager
 Developed by Danga (People behind memcache)
 Open source C daemon
 Native PHP extension
 Persistence – drizzle, postgres, memcache, sqllite
 Distributed
 No single point of failure
Beanstalkd
 Open source C daemon
 Fast
 Persistence – bin log file
 Distributed
 PHP library – pheanstalk, beanstalk
 Simple protocol
 Simple setup

Job
Client script Server Worker script

DISTRIBUTED SETUP
Beanstalkd

PUT Reserve DELETE

• Ready • Reserve • Poof

d
Beanstalkd client
Beanstalkd Worker
Beanstalkd Goodness
 Named Tubes – max 200 bytes
 Priority
 Delayed job
 Bury
 Kick
Advantages
 Speed up work
 Parallel and asynchronous work
 Scales easily
 Call functionality in other programming language
 Does not block your apache process.
Running worker script
 Use supervisord
 Gearman Manager
 Gear up
 Check for memory leaks in worker scripts
 Restart your worker scripts after certain number of
runs.
Questions
Abhinav Lal
Blog: abhinavlal.wordpress.com
Twitter: @abhinavlal
Slideshare: slideshare.net/abhinavlal