Chapter 11 Flashcards - News Feed System
flashcards volume1 news-feed social-media fanout cache
What are the two core operations in a news feed system?
?
1. PUBLISH POST (write path): User creates post → stored in Post DB → Fanout Service distributes post_id to follower feed caches → Notification Service alerts followers. 2. RETRIEVE FEED (read path): User opens app → News Feed Service fetches pre-computed feed from Redis cache → hydrate post_ids to full posts from Content Cache → merge with celebrity posts → return sorted feed. Keep these two paths completely separate for independent scaling.
What are the scale numbers for a typical news feed interview question?
?
10M DAU (Daily Active Users), max 500 friends per user (Facebook-style mutual friends), 10:1 read-to-write ratio (read-heavy system). Posts can contain text, images, videos. Feed is sorted reverse chronologically (newest first). Slight delay in feed freshness is acceptable (eventual consistency). Feed load target < 200ms.
What is fanout on write (push model) — how it works, pros, and cons?
?
How: When user publishes a post, immediately write post_id to each follower’s feed cache (Redis ZADD). Feed read = O(1) cache lookup. Pros: Extremely fast reads (pre-computed), real-time updates, simple read path. Cons: Celebrity problem — user with 10M followers triggers 10M cache writes per post (write amplification), wasted computation for inactive users who never read the pre-computed feed. Best for regular users with small follower counts.
What is fanout on read (pull model) — how it works, pros, and cons?
?
How: When user publishes, only store in Post DB (no fanout). On feed read: query friend list, fetch each friend’s recent posts, merge + sort by timestamp. Pros: No wasted computation (only compute when user reads), handles celebrities gracefully (no write amplification on post), feed is always perfectly fresh. Cons: Slow reads — N DB queries per feed fetch (N = number of friends), expensive read path under high concurrency, spiky load on Post DB. Best for celebrities or small-scale systems.
What is the hybrid fanout model and why is it the industry standard?
?
Combines fanout on write for regular users and fanout on read for celebrities. Regular users (≤ threshold followers, e.g., 1000): fanout on write — push post_id to all followers’ feed caches immediately. Celebrities (> threshold): skip fanout, just store post in DB. At feed read time, followers fetch celebrity posts directly and merge with pre-computed cache feed by timestamp. Used by Facebook, Twitter, Instagram. Balances fast reads with manageable write cost.
What is the celebrity problem and how is it solved?
?
Problem: A celebrity with 10M followers posts → naive fanout on write requires 10M Redis ZADD operations → write storm, takes minutes, overloads Redis. Solution: Define celebrity threshold (e.g., > 1000 followers). For celebrities: skip fanout entirely, mark as celebrity post in DB. At read time: fetch celebrity’s recent posts directly from Post DB/cache, merge with follower’s pre-computed feed by timestamp. Threshold is tunable based on system capacity.
Why is Redis Sorted Set (ZSET) the ideal data structure for news feed storage?
?
Sorted set stores members with a score (score = Unix timestamp). Advantages: 1) ZADD inserts in O(log N) with automatic ordering, 2) ZREVRANGE returns newest-first feed in O(log N + K), 3) ZREMRANGEBYRANK trims old posts to cap memory (keep last 500), 4) ZREM removes deleted posts, 5) Can merge celebrity posts with pre-computed feed by timestamp at read time. A list can’t do efficient timestamp-based merging. A hash can’t do range queries.
Write the Redis commands used to maintain and read a user’s news feed.
?
# Fanout: add post to feed (score = timestamp)
ZADD feed:{user_id} {timestamp} {post_id}
# Read latest 20 posts (newest first):
ZREVRANGE feed:{user_id} 0 19 WITHSCORES
# Cursor-based pagination (posts older than cursor):
ZREVRANGEBYSCORE feed:{user_id} {cursor-1} -inf LIMIT 0 20
# Trim feed to last 500 posts (save memory):
ZREMRANGEBYRANK feed:{user_id} 0 -501
# Delete a post from feed:
ZREM feed:{user_id} {post_id}
Score = Unix timestamp. Higher score = newer post = appears first.
What are the 5 cache layers in a well-designed news feed system?
?
- News Feed Cache: Redis ZSET per user (feed:{user_id} → sorted post_ids by timestamp). 2. Content Cache: Redis HashMap per post (post:{post_id} → full post data). 3. Social Graph Cache: Redis List per user (friends:{user_id} → friend ID list). 4. Action Cache: Redis counters (likes:{post_id} → like count, comments:{post_id} → count). 5. Counter Cache: Redis counters (followers:{user_id} → count, following:{user_id} → count). Each has its own TTL and invalidation strategy.
How does feed retrieval work when using the hybrid fanout model?
?
- Fetch pre-computed feed from Redis ZSET: feed:{user_id} (post_ids from regular friends). 2. Query social graph for celebrities this user follows. 3. Fetch each celebrity’s recent posts from Post DB or celebrity post cache. 4. Merge pre-computed post_ids + celebrity post_ids, sort by timestamp (newest first). 5. Take top N results. 6. Hydrate post_ids to full post objects from Content Cache (Redis HashMap). 7. Return hydrated feed to client. Steps 2-4 are the read-time fanout for celebrities.
How does cursor-based pagination work for news feeds, and why not use offset-based?
?
Cursor-based: Cursor = timestamp of the oldest post on current page. Next page: ZREVRANGEBYSCORE feed:{user_id} {cursor-1} -inf LIMIT 0 20. Why not offset? Feed updates continuously as new posts are added. Offset-based pagination (LIMIT 20 OFFSET 40) skips/duplicates posts when new items are inserted between pages. Cursor anchors to a specific point in time, giving stable pagination regardless of new inserts. Always use cursor-based for infinite scroll feeds.
How does the Fanout Service work with message queues?
?
Publishing a post is synchronous (save to Post DB), but fanout is async via message queue. Flow: Post Service → publishes event to Kafka topic “new_post” → Fanout Workers consume events → for each event, look up follower list → ZADD post_id to each follower’s Redis feed cache. Benefits: 1) Decouples post publishing from fanout (faster API response), 2) Absorbs fanout spikes, 3) Worker failures don’t lose events (Kafka retains messages), 4) Scale workers independently based on queue lag.
What data model stores the social graph (friend/follower relationships)?
?
Friendship (bidirectional, Facebook-style): Table friendships(user_id_1, user_id_2, created_at) — store one row per pair. Query: SELECT user_id_2 FROM friendships WHERE user_id_1 = X UNION SELECT user_id_1 FROM friendships WHERE user_id_2 = X. Follows (one-directional, Twitter-style): Table follows(follower_id, followee_id, created_at). Query: SELECT followee_id FROM follows WHERE follower_id = X. Both are cached in Social Graph Cache (Redis List, TTL 10 min) to avoid repeated DB queries.
How do you handle post deletion from news feeds at scale?
?
Soft delete in Post DB: Set deleted_at timestamp (never physically remove). Lazy removal from feed caches: Don’t iterate all follower caches to ZREM — too expensive for popular users. Instead: when hydrating post_ids to full posts (Content Cache), filter out posts where deleted_at is set. The post_id stays in feed cache but is invisible. Periodic background job can clean stale entries. For media: invalidate CDN cache entries for deleted media URLs. Trade-off: slightly stale cache vs expensive eager deletion.
How does the news feed system handle ranking (non-chronological feeds)?
?
Ranking signals: recency (time since posted), engagement (likes, comments, shares), affinity (how close is this friend — interaction history). Ranking formula: score = recency×0.5 + engagement×0.3 + affinity×0.2. Applied at read time (not fanout time) because engagement signals change after post — posts accumulate likes/comments over hours. Feed cache stores posts by timestamp; ranking reorders them at read time before returning to client. ML-based ranking (like Facebook’s EdgeRank) is a follow-up discussion.
How do you store and scale post data (the Post DB)?
?
Use MySQL (or Postgres) sharded by post_id. Schema: posts(post_id BIGINT PK, author_id BIGINT, content TEXT, media_urls JSON, visibility ENUM, created_at TIMESTAMP, deleted_at TIMESTAMP). Shard by post_id using consistent hashing so posts distribute evenly. Use composite index on (author_id, created_at DESC) for celebrity fanout-on-read queries. Store media (images/videos) separately in S3, reference via URL in media_urls. Serve media via CDN (CloudFront).
How does the content cache work and what does it store?
?
Content Cache (Redis) maps post_id to full post object, avoiding DB lookups per post. Key: post:{post_id}, Value: JSON blob {post_id, author_id, content, media_urls, created_at, like_count, comment_count}. Set on write (cache-aside or write-through). TTL: 24-48 hours (posts viewed most heavily in first 24 hours). On cache miss: fetch from Post DB, populate cache. Like counts and comment counts are updated frequently — fetch from Action Cache (separate Redis key) rather than embedding in post cache to avoid constant invalidation.
How do you handle the thundering herd when a celebrity posts and millions of followers try to read the post at once?
?
Problem: Celebrity posts, followers get notified, all open app simultaneously → millions of requests for same celebrity’s posts → Post DB overwhelmed. Solutions: 1) Cache celebrity posts in Content Cache aggressively (warm cache on post publish), 2) Use CDN for static media, 3) Read replica for Post DB, 4) Local in-process cache in News Feed Service (short TTL, 1-5 sec) to absorb burst reads for same post, 5) Request coalescing: if 1000 requests for post_id=X arrive simultaneously, only one DB query is made, result shared to all 1000.
What makes news feed a read-heavy system and how does the design reflect that?
?
Read-to-write ratio is ~10:1 or higher (users scroll feeds far more than they post). Design reflects this by: 1) Pre-computing feeds (fanout on write) so reads are O(1) cache hits, 2) Five layers of cache absorbing the majority of read traffic, 3) Using read replicas for DB, 4) CDN for media — never hit origin for images/videos, 5) News Feed Service is stateless + horizontally scaled behind load balancer. Writes (post publishing, fanout) are async and best-effort from user’s perspective.
What are the pros and cons of a pure chronological feed vs engagement-based ranking?
?
Chronological — Pros: simple to implement (sort by timestamp), predictable for users, easy to debug, consistent (same post always at same position). Cons: misses relevance (unimportant posts from recent activity rank above highly-engaging old posts), doesn’t maximize user engagement. Engagement-based — Pros: more engaging feeds (users see what matters), maximizes time-on-site. Cons: complex ML model to train, filter bubble effects, opaque to users, posts can get “buried” and never seen, requires continuous model updates. Real systems use engagement ranking (Facebook EdgeRank, Twitter timeline algorithm).
How does the system ensure availability of the news feed if the News Feed Cache (Redis) goes down?
?
Redis cluster with replication (primary + replica per shard). If primary fails: automatic failover to replica (Redis Sentinel or Redis Cluster). If entire cache cluster is unavailable: fall back to fanout on read for all users — fetch friend posts directly from Post DB (slower but correct). News Feed Service must handle cache misses gracefully (always have a DB fallback). Use circuit breaker: if cache error rate exceeds threshold, bypass cache and go directly to DB temporarily. Monitor Redis health continuously.
How does the system handle users with different friend counts efficiently?
?
Users with few friends (< 100): Fanout on write is very fast (100 ZADD ops). At read time, feed is instant. Users with many friends (500): Fanout on write takes 500 ZADD ops — still fast (~50ms with pipelining). Power users at threshold (~1000): Hybrid kicks in. Celebrities (> 1000 followers): No fanout on write, merge at read time. The 500-friend limit (Facebook model) keeps fanout on write manageable for all regular users. For Twitter (no friend limit), the celebrity threshold is more critical.
How do you implement “friends of friends” or “suggested posts” in a news feed?
?
This is a graph traversal problem. For friends-of-friends: BFS with depth limit 2 on social graph. Cache results (social graph is slow to traverse). Suggested posts algorithm: 1) Find users with high affinity score (mutual friends, shared interests), 2) Fetch their high-engagement posts, 3) Inject into feed with lower priority than direct friend posts. Implemented as a separate recommendation service feeding into the feed merger. Start simple (direct friends only) in interview, mention extensibility to social discovery as a follow-up.
How does media (images, videos) work in the news feed system?
?
Upload flow: Client → API Server → upload media to S3 (object store) → get S3 URL → store URL in post.media_urls. Serving flow: Client requests media → CDN edge node → if cached: serve from CDN (fast), if miss: CDN fetches from S3 → CDN caches → serves to client. Never serve media directly from application servers. CDN reduces latency and offloads origin. Video requires additional processing: transcoding service generates multiple resolutions (360p, 720p, 1080p) asynchronously before URLs are published in the post.
What is the difference between a Facebook-style news feed (friends) and a Twitter-style feed (followers)?
?
Facebook (friends): Bidirectional relationship — both must accept. Max ~5000 friends. Read your friends’ posts. Feed is personalized to mutual connections. Simpler graph traversal. Twitter (followers): One-directional — follow without approval. No follower limit (celebrities have millions). Read posts from anyone you follow. Celebrity problem is more severe. Fanout on write is riskier — must use hybrid model aggressively. In interview: clarify which model, as it drives the celebrity threshold and fanout strategy significantly.
Total Cards: 25
Review Time: 20-25 minutes
Priority: HIGH - Very common interview question, especially at Meta/Twitter!
Last Updated: 2026-04-13