Disclaimer: This article was written few years ago and may no longer be relevant as software engineering has changed a lot in the last few years. This is what may be more relevant now: Future of Software Engineering - Gaurav ChandakFollowing Engineering Blogs is one of the best ways to understand how the engineering teams at the top tech companies function and how they build scalable systems.
It also helps in preparing for System Design Interviews as you get to understand the trade-offs that these companies took in making their architecture scalable based on their requirements.
I have curated a list of the top engineering blogs and have also added the recommended articles from each of them to help you learn more about their architecture and prepare for System Design (HLD) Interviews.
I would highly suggest you to go through 1-2 articles/videos everyday. It would hardly take 30 mins/day. Consistency is key.
You do not need to completely understand everything. Just a brief idea about what's happening is good enough. Things will start making sense once you've read a lot of the articles. If you find something interesting and would like to learn more then Google to get more information.
Netflix TechBlog
Link: Netflix TechBlog
Netflix has one of the best tech blogs in the industry. They have written a lot about their learnings and practices across a bunch of different areas. Their blog and open source infra has helped a lot of companies scale and adopt Netflix's engineering practices. They have also been pioneers in the adoption of microservices architecture across the industry.
Recommended Links:
- Overview
- Mastering Chaos - A Netflix Guide to Microservices
- Microservices at Netflix Scale: Principles, Tradeoffs & Lessons Learned
- How Netflix works: the (hugely simplified) complex stuff that happens every time you hit Play
- Microservices
- Netflix Conductor: A microservices orchestrator
- Evolution of Netflix Conductor: v2.0 and beyond
- Engineering Trade-Offs and The Netflix API Re-Architecture
- Fault Tolerance and Resiliency
- Fault Tolerance in a High Volume, Distributed System
- Keeping Netflix Reliable Using Prioritized Load Shedding
- Making the Netflix API More Resilient
- Chaos Engineering
- Streaming
- Content Popularity for Open Connect
- Distributing Content to Open Connect
- Scaling Event Sourcing for Netflix Downloads, Episode 1
- Scaling Event Sourcing for Netflix Downloads, Episode 2
- High Quality Video Encoding at Scale
- Scaling Time Series Data Storage — Part I
- Scaling Time Series Data Storage — Part II
- Data Compression for Large-Scale Streaming Experimentation
- Architecuture of other products
- The Netflix Media Database
- Modernizing the Web Playback UI
- Building a Cross-platform In-app Messaging Orchestration Service
- Building a Rule-Based Platform to Manage Netflix Membership SKUs at Scale
- Creating a Scalable Offers Platform
- Common Infrastructure
- Netflix Edge Authentication & Identity
- Keystone Real-time Stream Processing Platform
- Rethinking Netflix's Edge Load Balancing
- Caching for a Global Netflix
- Cache warming: Agility for a stateful service
- Distributed delay queues based on Dynomite
- Scalable Logging and Tracking
- Recommendation Engine
Uber Engineering Blog
Link: Uber Engineering Blog
Recommended Links:
- Engineering the Architecture Behind Uber’s New Rider App
- Uber's Real-Time Push Platform
- Engineering Uber's Next-Gen Payments Platform
- Service-Oriented Architecture
- Rewriting Uber Engineering: The Opportunities Microservices Provide
- Why We Leverage Multi-tenancy in Uber's Microservice Architecture
- Introducing Domain-Oriented Microservice Architecture
- The Architecture of Uber's API gateway
Airbnb Engineering Blog
Link: Infrastructure at Airbnb
Recommended Links:
- Building Services at Airbnb, Part 1
- Building Services at Airbnb, Part 2
- Building Services at Airbnb, Part 3
- Building Services at Airbnb, Part 4
- Avoiding Double Payments in a Distributed Payments System
- Measuring Transactional Integrity in Airbnb’s Distributed Payment Ecosystem
- Airbnb's Promotions and Communications Platform
- Himeji: a scalable centralized system for authorization at Airbnb
- Dynein: Building an Open-source Distributed Delayed Job Queueing System
- Scaling Knowledge Access and Retrieval at Airbnb
Hotstar Tech Blog
Hotstar has one of the top tech blogs in India.
Link: Hotstar Tech Blog
Recommended Links:
- Scaling the Hotstar Platform for 50M
- Re-Architecting Apps For Scale
- Service Monitoring Patterns
- Capturing A Billion Emo(j)i-ons
- Scaling Subscriptions : Cache for Cash
- “Millons scale” simulations
- T For Tsunami : Dealing with traffic spikes
- Ingesting data at “Bharat” Scale
- Data Democratisation @ Hotstar
- Building Pubsub for 50M concurrent socket connections
- Read about our experiments with localised and personalised artwork.
- Engineering for 1,000,000 requests per second using Golang
- Building a scalable data platform at Hotstar
Myntra Engineering Blog
Myntra has been at the forefront of innovation in the e-commerce space in India. Their engineering blog provides a lot of insights regarding the same.
Link: Myntra Engineering
Recommended Links:
- How to build a technology platform
- Decoding user queries in E-Commerce
- Adaptive Throttling of Indexing for Improved Query Responsiveness
- Real-time context-based smart type-ahead suggestions
- Semantic Search — Innovation at scale!
- Discount offers using Dynamic Programming?
- Vulcan: Inventory Redistribution at Myntra
- Myntra Scheduler Service -Fault-tolerant, horizontally scalable managed scheduler service
- Realtime Big Data Ingestion with Meterial
Swiggy Tech Blog
Link: Swiggy Bytes
Recommended Links:
- A brief introduction to Engineering challenges at Swiggy
- Re-Architecting Swiggy's logistics systems
- What Serviceability means at Swiggy?
- The Swiggy Delivery Challenge (Part One)
- The Swiggy Delivery Challenge (Part Two)
- Designing the Serviceability Platform at Swiggy for High Scale — Part 1
- Building Blocks of Swiggy Stores — From Food Delivery to a Hyper-Local Marketplace
- Swiggy Distance Service
- Kabootar — Swiggy's Communication Platform
- Swiggylytics: Swiggy's real-time Analytics SDK
- Evolution of and experiments with feed ranking at Swiggy
- Chatbots at Swiggy
- How to identify and quantify ‘Consumer Love’ to drive business outcomes?
- Personalizing Swiggy POP Recommendations
- Running Geo Queries At Scale
Flipkart Tech Blog
Link: Flipkart Tech Blog
Recommended Links:
- Scaling and building platform resilience
- Latency and Fault tolerance in OLTP @ 1.5 billion/day service calls
- Scaling write-heavy OLTP systems with strong data guarantees
- Lego Building Blocks To Model Supply Chain Workflows
- Flipkart Data Platform — India's largest eCommerce Big Data Platform
- Services Fulfilment Cockpit
- Foxtrot — Event Analytics At Scale
- Proxies for resilience and fault tolerance in distributed SOA
- Making deliveries faster — The Flipkart Cache Cluster
- Choosing a datastore for the Flipkart User Engagement Platform
- Predicting your next query even before you type!
- Building Personalized Autosuggestion in Flipkart
- What's in a Query?
- Adapting Search to Indian Phonetics
- Just in Time Personalization
- Aspect Ratings & Reviews
- Conversational Assistant Platform- changing the way India shops online
Gojek Tech Blog
Link: Gojek Tech Blog
Recommended Link:
- How we define speed, execution and scale
- Why We Swear by the RCA
- Resiliency in Distributed Systems
- How to build Resilience in large scale Distributed Systems
- How GO-JEK handles microservices communication at scale
- Applying the Single Responsibility Principle to Microservices
- Reducing Latency with Sidecar
- Data infrastructure at GO-JEK
- The Journey to Our New Authentication System
- How We Manage a Million Push Notifications an Hour
- On Concurrency Control in Databases
- How we built 'BARITO' to enhance logging
- Fronting : An Armoured Car for Kafka Ingestion
- How to Unlock the Full Potential of Kafka Producers
- How we built 'Global Search' to improve discovery
- Hospital — Our Automated Solution for System Failures
- How We Supercharged Chat
- Introducing GO-JEK's card personalization engine: Project shuffle
- How We Built a Matchmaking Algorithm to Cross-Sell Products
Grab Tech Blog
Link: Grab Tech
Recommended Links:
- Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)
- Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)
- Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting
- Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks
- Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering
- How We Prevented App Performance Degradation from Sudden Ride Demand Spikes
- How We Scaled Our Cache and Got a Good Night's Sleep
- Scaling Like a Boss with Presto
- Serving Driver-partners Data at Scale Using Mirror Cache
- Pharos - Searching Nearby Drivers on Road Network at Scale
- Customer Support Workforce Routing
- How We Built Our In-house Chat Platform for the Web
- Data First, SLA Always
- How We Simplified Our Data Ingestion & Transformation Process
- Trident - Real-time Event Processing at Scale
- A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform
- How We Built a Logging Stack at Grab Problem
- Structured Logging: The Best Friend You'll Want When Things Go Wrong
- Optimally Scaling Kafka Consumer Applications
- Plumbing At Scale
- Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily
- Building Grab's Experimentation Platform
- Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab
- Orchestrating Chaos Using Grab's Experimentation Platform
- Round-robin in Distributed Systems
- Grab You Some Post-Mortem Reports
- Introducing Grab-Kit: Distributed Service Design at Grab
Discord Engineering Blog
Link: Discord Engineering Blog
Recommended Links:
- How Discord Stores Billions of Messages
- How Discord Indexes Billions of Messages
- How Discord Handles Two and Half Million Concurrent Voice Users using WebRTC
- How Discord Scaled Elixir to 5,000,000 Concurrent Users
- How Discord Resizes 150 Million Images Every Day with Go and C++
- How Discord handles push request bursts of over a million per minute with Elixir’s GenStage
Twitter Engineering Blog
Recommended Links:
- Operations at Twitter: Scaling Beyond 100 Million Users
- Twitter The Infrastructure Behind Twitter: Scale
- The infrastructure behind Twitter: efficiency and optimization
- #ExpandTheEdge: Making Twitter Faster
- Rebuilding Twitter's public API
- Twitter: New Tweets per second record, and how!
- Snowflake - Unique UUID
- Building Twitter's ad platform architecture for the future
- How we fortified Twitter's real time ad spend architecture
- Storing and retrieving millions of ad impressions per second at Twitter
- Sharding, simplification, and Twitter's ads serving platform
- Twitter: Streaming logging pipeline of Home timeline prediction system
- Deterministic Aperture: A distributed, load balancing algorithm
- Twitter MetricsDB: TimeSeries Database for storing metrics at Twitter
- "Introducing FlockDB | Twitter Blogs." Introducing FlockDB
- Twitter's Kafka adoption story
- Kafka as a storage system (blog.twitter.com)
- Dynamic configuration at Twitter
- Building DistributedLog: High-performance replicated log service
Instagram Engineering Blog
Link: Instagram Engineering
Recommended Links:
- What Powers Instagram: Hundreds of Instances, Dozens of Technologies
- Making Instagram.com faster: Part 1
- Making Instagram.com faster: Part 2
- Making Instagram.com faster: Part 3 — cache first
- Making instagram.com faster: Code size and execution optimizations (Part 4)
- Search Architecture at Instagram
- Improving Instagram's Music Audio Quality
- Making Direct Messages Reliable and Fast | by Tommy Crush
- Sharding & IDs at Instagram
- Instagram: Storing hundreds of millions of simple key-value pairs in Redis
Facebook, Whatsapp and YouTube
Recommended Links:
- Facebook and memcached - Mark Zuckerberg
- The Storage Technologies Behind Facebook Messages
- Messaging architecture at Facebook
- Building Real Time Infrastructure at Facebook
- Scaling to Millions of Simultaneous Connections
- A Look at WhatsApp: Engineering for Success at Scale
- How WhatsApp Reduced Spam for Over 1 Billion People
- YouTube Scalability
LinkedIn Engineering Blog
Link: LinkedIn Engineering Blog
Recommended Links:
- An inside look at LinkedIn’s data pipeline monitoring system
- Related story InSearch: LinkedIn's new message search platform
- Rebuilding messaging: How we designed our new system
- Rebuilding messaging: How we bootstrapped our platform
- Rebuilding messaging: How we built for extensibility
- Building a more intuitive and streamlined search experience
Pinterest Engineering Blog
Link: Pinterest Engineering Blog
Recommended Links:
- Scaling Pinterest
- Building a smarter home feed
- Building a scalable and available home feed
- Geo-blocking media content on Pinterest
- Optimizing video playback performance
- Rebuilding the user typeahead
- Improving search relevance and engagement with text attributes
- Manas: A high performing customized search system
- Manas Realtime — Enabling changes to be searchable in a blink of an eye
- Building a real-time user action counting system for ads
- How Pinterest runs Kafka at scale
- Optimizing Kafka for the cloud. By Ambud Sharma | Software Engineer…
- Scalable and reliable data ingestion at Pinterest
- Sharding Pinterest: How we scaled our MySQL fleet
- Goku: Building a scalable and high performant time-series database system
- Building Pinalytics: Pinterest's data analytics engine
- Real-time analytics at Pinterest
Dropbox Engineering Blog
Link: Infrastructure - Dropbox
Recommended Links:
- How We've Scaled Dropbox
- Streaming File Synchronization
- Video Processing at Dropbox
- Instant, Full-Text Search Engine for Dropbox
- Improving the performance of full-text search
- Optimizing web servers for high throughput and low latency
- Improving Dropbox Performance: Retrieving Thumbnails
- Cross shard transactions at 10 million requests per second
- Pushing the edges of our global performance
- Evolution of Dropbox's Edge Network
- Evolution of the Dropbox backbone network
- Dropbox traffic infrastructure: Edge network
- Intelligent DNS based load balancing at Dropbox
Stripe Engineering Blog
Link: Stripe Engineering Blog
Recommended Links:
- Designing robust and predictable APIs with idempotency
- Scaling your API with rate limiters
- APIs as infrastructure: future-proofing Stripe with versioning
I hope that you found this article to be extremely insightful. If you feel that I've missed some blog/article/video then do ping me on Whatsapp (
