diff --git a/README.md b/README.md index 4344fef..3b810e3 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,80 @@ # system-design-resources These are the best resources for System Design on the Internet + +Transcoding Videos at Scale: https://www.egnyte.com/blog/2018/12/transcoding-how-we-serve-videos-at-scale/ +AirBnb Idempotency: https://medium.com/airbnb-engineering/avoiding-double-payments-in-a-distributed-payments-system-2981f6b070bb +Facebook Cluster Management: https://engineering.fb.com/data-center-engineering/twine/ +Google Autopilot - Autoscaling: https://dl.acm.org/doi/pdf/10.1145/3342195.3387524 +Netflix Workflow Orchestration: https://netflix.github.io/conductor/ +Opensource Workflow Management: https://github.com/spotify/luigi +Facebook Video Broadcasting: https://engineering.fb.com/ios/under-the-hood-broadcasting-live-video-to-millions/ +LinkedIn Brooklin- Real time data streaming: https://engineering.linkedin.com/blog/2019/brooklin-open-source +Amazon S3 Performance hacks: https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/ +Amazon S3 object expiration: https://aws.amazon.com/blogs/aws/amazon-s3-object-expiration/ +Open Source Distributed File System: https://docs.ceph.com/en/latest/architecture/ +Pintrest Time Series Database: https://medium.com/pinterest-engineering/goku-building-a-scalable-and-high-performant-time-series-database-system-a8ff5758a181 +Uber Time Series DB: https://eng.uber.com/aresdb/ +TimeSeries Relational DB: https://blog.timescale.com/blog/time-series-data-why-and-how-to-use-a-relational-database-instead-of-nosql-d0cd6975e87c/ +Facebook Gorilla Time Series DB: http://www.vldb.org/pvldb/vol8/p1816-teller.pdf +Circuit Breaker Algorithm: https://martinfowler.com/bliki/CircuitBreaker.html +Uber Rate Limiter: https://github.com/uber-go/ratelimit/blob/master/ratelimit.go +What is HTTP: https://engineering.cred.club/head-of-line-hol-blocking-in-http-1-and-http-2-50b24e9e3372 +QUIC Protocol: https://www.akamai.com/blog/performance/http3-and-quic-past-present-and-future +Chess Engine Building: https://www.youtube.com/watch?v=U4ogK0MIzqk +Subscription Manager: https://netflixtechblog.com/building-a-rule-based-platform-to-manage-netflix-membership-skus-at-scale-e3c0f82aa7bc +Operational Transform: http://www.codecommit.com/blog/java/understanding-and-applying-operational-transformation +Google Docs: https://www.youtube.com/watch?v=uOFzWZrsPV0&list=PLXDe3d8o9VFtydBV5biyz9iS3WqKsBMD5&index=3 +Uber Distributed Request Tracing: https://eng.uber.com/distributed-tracing/ +Elastic Search Time Based Querying: https://www.elastic.co/guide/en/elasticsearch/guide/current/time-based.html +Elastic Search Aggregation: https://www.elastic.co/guide/en/elasticsearch/guide/current/aggregations.html +Pintrest Logging: https://medium.com/@Pinterest_Engineering/open-sourcing-singer-pinterests-performant-and-reliable-logging-agent-610fecf35566 +Google Monitoring Infrastructure: https://www.facebook.com/atscaleevents/videos/959344524420015/ +Kubernetes Service Mesh: https://akomljen.com/kubernetes-service-mesh/ +Kubernetes Sidecar:https://www.weave.works/blog/introduction-to-service-meshes-on-kubernetes-and-progressive-delivery +Nginx Service Mesh: https://www.nginx.com/learn/service-mesh/ +Facebook Data Engineering: https://engineering.fb.com/2016/05/09/core-data/introducing-fblearner-flow-facebook-s-ai-backbone/ +Amazon Dynamo DB Internals: https://www.allthingsdistributed.com/2007/10/amazons_dynamo.html +Indexing in Cassandra: https://www.bmc.com/blogs/cassandra-clustering-columns-partition-composite-key/ +Design Patterns in Amazon Dynamo DB: https://www.youtube.com/watch?v=HaEPXoXVf2k +Internals of Amazon Dynamo DB: https://www.youtube.com/watch?v=yvBR71D0nAQ +API Design: https://medium.com/airbnb-engineering/building-services-at-airbnb-part-1-c4c1d8fa811b +Swagger APIs: https://swagger.io/docs/specification/about/ +Sorted String Tables and Compaction Strategies: https://github.com/scylladb/scylla/wiki/SSTable-compaction-and-compaction-strategies +Hyperloglog Algorithm: https://odino.org/my-favorite-data-structure-hyperloglog/ +Caching: http://highscalability.com/blog/2016/1/25/design-of-a-modern-cache.html +Microsoft Caching Guide: https://docs.microsoft.com/en-us/previous-versions/msp-n-p/dn589802(v%3dpandp.10) +Database replication: https://dev.mysql.com/doc/refman/8.0/en/replication.html +Capacity Estimation: http://static.googleusercontent.com/media/research.google.com/en//people/jeff/stanford-295-talk.pdf +Facebook Twine Containerization: https://engineering.fb.com/developer-tools/zookeeper-twine/ +CloudFlare Containerization: https://blog.cloudflare.com/cloud-computing-without-containers/ +Docker Architecture: https://docs.docker.com/get-started/overview/#docker-architecture +Monolith to Microservice: https://martinfowler.com/articles/break-monolith-into-microservices.html +Netflix Data replication - Change Data Capture: https://netflixtechblog.com/dblog-a-generic-change-data-capture-framework-69351fb9099b +YouTube Architecture: http://highscalability.com/youtube-architecture +YouTube scalability 2012: https://www.youtube.com/watch?v=w5WVu624fY8 +Google Capacity Estimation: https://www.youtube.com/watch?v=modXC5IWTJI +Scalability at YouTube 2012: https://www.youtube.com/watch?v=G-lGCC4KKok +Back of envelope Calculations at AWS: https://www.youtube.com/watch?v=-3qetLv2Yp0 +Oracle Publisher Subscriber: https://docs.oracle.com/cd/B10501_01/appdev.920/a96590/adg15pub.htm +Amazon Pub Sub Messaging: https://aws.amazon.com/pub-sub-messaging/ +Asynchronous processing: http://blog.codepath.com/2013/01/06/asynchronous-processing-in-web-applications-part-2-developers-need-to-understand-message-queues/ +Async Request Response: https://www.enterpriseintegrationpatterns.com/patterns/conversation/RequestResponse.html +DB as queue Antipattern: http://blog.codepath.com/2012/11/15/asynchronous-processing-in-web-applications-part-1-a-database-is-not-a-queue/ +Using a database as a message queue: https://softwareengineering.stackexchange.com/questions/231410/why-database-as-queue-so-bad +Anti-pattern of DB as a queue: http://mikehadlow.blogspot.com/2012/04/database-as-queue-anti-pattern.html +Drawbacks of DB as a queue: https://www.cloudamqp.com/blog/why-is-a-database-not-the-right-tool-for-a-queue-based-system.html +Oracle Single Points of failure: https://docs.oracle.com/cd/E19693-01/819-0992/fjdch/index.html +DNS single point of failure 2004: http://www.tenereillo.com/GSLBPageOfShame.htm +Martin Fowler- Event Driven Architecture: https://www.youtube.com/watch?v=STKCRSUsyP0 +Event Driven Architecture: https://martinfowler.com/articles/201701-event-driven.html +Monolith Architecture: https://buttercms.com/books/microservices-for-startups/should-you-always-start-with-a-monolith/ +Monoliths vs Microservices: https://articles.microservices.com/monolithic-vs-microservices-architecture-5c4848858f59 +Microservices: http://highscalability.com/blog/2018/4/5/do-you-have-too-many-microservices-five-design-attributes-th.html +Uber Nanoservices antipattern: https://www.youtube.com/watch?v=kb-m2fasdDY +Load Balancer with Sticky Sessions: https://stackoverflow.com/questions/10494431/sticky-and-non-sticky-sessions +Citrix what is load balancing: https://www.citrix.com/en-in/solutions/app-delivery-and-security/load-balancing/what-is-load-balancing.html +Nginx Load Balancing: https://www.nginx.com/resources/glossary/load-balancing/ +What is a message queue: https://www.cloudamqp.com/blog/what-is-message-queuing.html +Distributed Design Patterns: http://horicky.blogspot.com/2010/10/scalable-system-design-patterns.html +Consistent hashing: https://michaelnielsen.org/blog/consistent-hashing/ +Facebook Messenger Optimisations: https://spectrum.ieee.org/how-facebooks-software-engineers-prepare-messenger-for-new-years-eve