Scribd Logo
Upload

Welcome to Scribd!

  • 216 views

    Redis at Superfeedr: in Production @superfeedr // @julien51

    Uploaded by

    Julien Genestoux
    Redis is used at Superfeedr for scheduling and caching in their real-time feed parsing system. A Redis list called the "ring" is used as a data structure that can be rotated to store and retrieve items with priorities to fetch. The algorithm pops items from the ring, gets any associated data from Redis, and fetches items if their time has arrived, otherwise puts them back, with over 450 processes running this loop to handle millions of items per second. Various optimizations are discussed including pipelining, sharding, and 32-bit compilation to improve performance on their servers.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PPT, PDF, TXT or read online from Scribd
    Flag for inappropriate content

    You might also like

    Redis at Superfeedr: in Production @superfeedr // @julien51

    Uploaded by

    Julien Genestoux
    216 views8 pages
    Redis is used at Superfeedr for scheduling and caching in their real-time feed parsing system. A Redis list called the "ring" is used as a data structure that can be rotated to store and retrieve items with priorities to fetch. The algorithm pops items from the ring, gets any associated data from Redis, and fetches items if their time has arrived, otherwise puts them back, with over 450 processes running this loop to handle millions of items per second. Various optimizations are discussed including pipelining, sharding, and 32-bit compilation to improve performance on their servers.

    Original Description:

    Redis at Superfeedr : the ring

    Original Title

    redis@superfeedr

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Available Formats

    PPT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    Redis is used at Superfeedr for scheduling and caching in their real-time feed parsing system. A Redis list called the "ring" is used as a data structure that can be rotated to store and retrieve items with priorities to fetch. The algorithm pops items from the ring, gets any associated data from Redis, and fetches items if their time has arrived, otherwise puts them back, with over 450 processes running this loop to handle millions of items per second. Various optimizations are discussed including pipelining, sharding, and 32-bit compilation to improve performance on their servers.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PPT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as ppt, pdf, or txt
    216 views8 pages

    Redis at Superfeedr: in Production @superfeedr // @julien51

    Uploaded by

    Julien Genestoux
    Redis is used at Superfeedr for scheduling and caching in their real-time feed parsing system. A Redis list called the "ring" is used as a data structure that can be rotated to store and retrieve items with priorities to fetch. The algorithm pops items from the ring, gets any associated data from Redis, and fetches items if their time has arrived, otherwise puts them back, with over 450 processes running this loop to handle millions of items per second. Various optimizations are discussed including pipelining, sharding, and 32-bit compilation to improve performance on their servers.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PPT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as ppt, pdf, or txt
    of 8

    Redis at Superfeedr

    In production
    @superfeedr // @julien51
    Superfeedr What?

    • Realtime feed parsing in the cloud


    • Polling feeds
    • Redis for
    • Scheduling
    • Caching
    The Ring

    • Requirements
    • A datastructure that can be “rotated”
    • Dealing with priorities
    • Storing some “basic” stuff
    Redis
    • List rpoplupsuh(“ring”, “ring”)
    • Key/Value in Json
    • Async
    • Pipelining to avoid network latencies
    • Version
    • redis_version:1.2.5
    • arch_bits:32
    • multiplexing_api:epoll
    Algorithm

    loop do
    ids = $redis.rpoplpush("ring", "ring", 50)
    items = $redis.get(ids)
    ids.each do |item|
    if item.time_to_fetch < now()
    fetch(item)
    item.time_to_fetch = later()
    else
    # skip
    end
    end
    end
    Cleaning the Ring

    • No direct access to items in List


    • ‘maintenance’ script that takes items in the
    back, check if they’re in k/v space and puts
    them back (or not) in the ring.
    Numbers
    • 450 processes with the same loop.
    • ~=2M items in ring in 15seconds (limited
    by clients perf)
    • 1.5GB slice @slicehost. RSS ~= 600MB
    • Total commands/sec : 102869
    • 4core : load average: 1.00, 1.00, 1.00
    Tricks
    • Don’t snapshot/AOF on same machine. Use
    slave for this
    • Compile in 32 bits : less RAM
    • Pipeline! (cut down overhead %)
    • Client side sharding (ouch!)
    • Runit for process monitoring

    You might also like