Realtime Information Processing At Facebook
Recently in that place has been a lot of evolution inward realtime information processing systems, including Twitter's Storm in addition to Heron, Google's Millwheel, in addition to LinkedIn's Samza. This newspaper presents Facebook's Realtime information processing scheme architecture in addition to its Puma, Swift, in addition to Stylus current processing systems. The newspaper is titled "Realtime Data Processing at Facebook" in addition to it appeared at Sigmod'16, June 26-July 1.
Another large application is the mobile analytics pipelines that render realtime feedback for Facebook mobile application developers, who usage this information to diagnose performance in addition to correctness issues.
Scribe plays a fundamental role inward Facebook's realtime processing architecture. The primary stance of the architecture is this: By trading seconds versus milliseconds latency, the architecture is able to employ a persistent message bus, i.e., Scribe, for information transport. Scribe provides a persistent, distributed messaging scheme for collecting, aggregating in addition to delivering high volumes of log information alongside a few seconds of latency in addition to high throughput. Scribe is the carry machinery for sending information to both batch in addition to realtime systems at Facebook. Using Scribe to decouple the information carry from the processing allows the scheme to compass error tolerance, scalability, in addition to ease of use, as good as supporting multiple processing systems as options.
While Scribe incurs a few seconds of latency, it withal meets Facebook's performance requirements for latency in addition to provides hundreds of Gigabytes per minute throughput. On the other hand, Scribe provides a persistent message coach service that enables decoupling in addition to isolation of the information production in addition to information analysis scheme components. Moreover, alongside persistent Scribe streams, the scheme tin replay a current from a recent fourth dimension period, which makes debugging in addition to iterative-development much easier.
The Kafka log weblog yesteryear Jay Kreps described these benefits nicely as well. It talked most how practical systems tin yesteryear simplified alongside a log-centric design, in addition to how these log steams tin enable information Integration yesteryear making all of an organization's information easily available inward all its storage in addition to processing systems. Kafka would have got similar advantages to Scribe. Facebook uses Scribe because it is developed inward house.
Below I re-create snippets of descriptions from the newspaper for each of these subsystems.
Within Scribe, information is organized yesteryear distinct streams of "category". Usually, a streaming application consumes i Scribe category as input. A Scribe category has multiple buckets. A Scribe bucket is the basic processing unit of measurement for current processing systems: applications are parallelized yesteryear sending dissimilar Scribe buckets to dissimilar processes. Scribe provides information durability yesteryear storing it inward HDFS. Scribe messages are stored in addition to streams tin hold upwards replayed yesteryear the same or dissimilar receivers for upwards to a few days.
The realtime current processing systems Puma, Stylus, in addition to Swift read information from Scribe in addition to also write to Scribe. Laser, Scuba, in addition to Hive are information stores that usage Scribe for ingestion in addition to serve dissimilar types of queries. Laser tin also render information to the products in addition to streaming systems, as shown yesteryear the dashed (blue) arrows.
Puma is a current processing scheme whose applications (apps) are written inward a SQL-like linguistic communication alongside UDFs (user-defined functions) written inward Java. Puma apps are quick to write: it tin have got less than an hr to write, test, in addition to deploy a novel app. Unlike traditional relational databases, Puma is optimized for compiled queries, non for ad-hoc analysis. Puma provides filtering in addition to processing of Scribe streams (with a few seconds delay). The output of these stateless Puma apps is some other Scribe stream, which tin in addition to then hold upwards the input to some other Puma app, whatever other realtime current processor, or a information store.
Swift is a basic current processing engine which provides checkpointing functionalities for Scribe. If the app crashes, you lot tin restart from the latest checkpoint; all information is thence read at to the lowest degree i time from Scribe. Swift is by in addition to large useful for depression throughput, stateless processing.
Stylus is a low-level current processing framework written inward C++. A Stylus processor tin hold upwards stateless or stateful. Stylus's processing API is similar to that of other procedural current processing systems.
Laser is a high question throughput, depression (millisecond) latency, key-value storage service built on top of RocksDB. Laser tin hold upwards used to brand the lawsuit of a complex Hive question or a Scribe current available to a Puma or Stylus app, normally for a lookup join, such as identifying the topic for a given hashtag.
Scuba is Facebook's fast slice-and-dice analysis information store, most commonly used for trouble-shooting of problems as they happen. Scuba provides promotion hoc queries alongside most answer times nether 1 second.
Hive is Facebook's exabyte-scale information warehouse. Facebook generates multiple novel petabytes of information per day, most one-half of which is raw lawsuit information ingested from Scribe. (The other one-half of the information is derived from the raw data, e.g., yesteryear daily question pipelines.) Most lawsuit tables inward Hive are partitioned yesteryear day. Scribe does non render interplanetary space retention; instead Facebook stores input in addition to output streams inward our information warehouse Hive for longer retention.
Figure iv summarizes the 5 blueprint decisions considered for this Facebook realtime processing scheme components. Figure 5 summarizes which alternatives were chosen yesteryear a multifariousness of realtime systems, both at Facebook in addition to inward the related literature.
The highlights from this department are as follows:
Holistic Configuration Management at Facebook
Facebook's Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services
Measuring in addition to Understanding Consistency at Facebook
Motivation in addition to applications
Facebook runs hundreds of realtime information pipelines inward productions. As a motivation of the realtime information processing scheme the newspaper gives Chorus as an example. The Chorus information pipeline transforms a current of private Facebook posts into aggregated, anonymized, in addition to annotated visual summaries. E.g., what are the top 5 topics existence discussed for the election today? What are the demographic breakdowns (age, gender, country) of World Cup fans?Another large application is the mobile analytics pipelines that render realtime feedback for Facebook mobile application developers, who usage this information to diagnose performance in addition to correctness issues.
The scheme architecture
Scribe plays a fundamental role inward Facebook's realtime processing architecture. The primary stance of the architecture is this: By trading seconds versus milliseconds latency, the architecture is able to employ a persistent message bus, i.e., Scribe, for information transport. Scribe provides a persistent, distributed messaging scheme for collecting, aggregating in addition to delivering high volumes of log information alongside a few seconds of latency in addition to high throughput. Scribe is the carry machinery for sending information to both batch in addition to realtime systems at Facebook. Using Scribe to decouple the information carry from the processing allows the scheme to compass error tolerance, scalability, in addition to ease of use, as good as supporting multiple processing systems as options.
While Scribe incurs a few seconds of latency, it withal meets Facebook's performance requirements for latency in addition to provides hundreds of Gigabytes per minute throughput. On the other hand, Scribe provides a persistent message coach service that enables decoupling in addition to isolation of the information production in addition to information analysis scheme components. Moreover, alongside persistent Scribe streams, the scheme tin replay a current from a recent fourth dimension period, which makes debugging in addition to iterative-development much easier.
The Kafka log weblog yesteryear Jay Kreps described these benefits nicely as well. It talked most how practical systems tin yesteryear simplified alongside a log-centric design, in addition to how these log steams tin enable information Integration yesteryear making all of an organization's information easily available inward all its storage in addition to processing systems. Kafka would have got similar advantages to Scribe. Facebook uses Scribe because it is developed inward house.
Below I re-create snippets of descriptions from the newspaper for each of these subsystems.
Within Scribe, information is organized yesteryear distinct streams of "category". Usually, a streaming application consumes i Scribe category as input. A Scribe category has multiple buckets. A Scribe bucket is the basic processing unit of measurement for current processing systems: applications are parallelized yesteryear sending dissimilar Scribe buckets to dissimilar processes. Scribe provides information durability yesteryear storing it inward HDFS. Scribe messages are stored in addition to streams tin hold upwards replayed yesteryear the same or dissimilar receivers for upwards to a few days.
The realtime current processing systems Puma, Stylus, in addition to Swift read information from Scribe in addition to also write to Scribe. Laser, Scuba, in addition to Hive are information stores that usage Scribe for ingestion in addition to serve dissimilar types of queries. Laser tin also render information to the products in addition to streaming systems, as shown yesteryear the dashed (blue) arrows.
Puma is a current processing scheme whose applications (apps) are written inward a SQL-like linguistic communication alongside UDFs (user-defined functions) written inward Java. Puma apps are quick to write: it tin have got less than an hr to write, test, in addition to deploy a novel app. Unlike traditional relational databases, Puma is optimized for compiled queries, non for ad-hoc analysis. Puma provides filtering in addition to processing of Scribe streams (with a few seconds delay). The output of these stateless Puma apps is some other Scribe stream, which tin in addition to then hold upwards the input to some other Puma app, whatever other realtime current processor, or a information store.
Swift is a basic current processing engine which provides checkpointing functionalities for Scribe. If the app crashes, you lot tin restart from the latest checkpoint; all information is thence read at to the lowest degree i time from Scribe. Swift is by in addition to large useful for depression throughput, stateless processing.
Stylus is a low-level current processing framework written inward C++. A Stylus processor tin hold upwards stateless or stateful. Stylus's processing API is similar to that of other procedural current processing systems.
Laser is a high question throughput, depression (millisecond) latency, key-value storage service built on top of RocksDB. Laser tin hold upwards used to brand the lawsuit of a complex Hive question or a Scribe current available to a Puma or Stylus app, normally for a lookup join, such as identifying the topic for a given hashtag.
Scuba is Facebook's fast slice-and-dice analysis information store, most commonly used for trouble-shooting of problems as they happen. Scuba provides promotion hoc queries alongside most answer times nether 1 second.
Hive is Facebook's exabyte-scale information warehouse. Facebook generates multiple novel petabytes of information per day, most one-half of which is raw lawsuit information ingested from Scribe. (The other one-half of the information is derived from the raw data, e.g., yesteryear daily question pipelines.) Most lawsuit tables inward Hive are partitioned yesteryear day. Scribe does non render interplanetary space retention; instead Facebook stores input in addition to output streams inward our information warehouse Hive for longer retention.
Design decisions
Figure iv summarizes the 5 blueprint decisions considered for this Facebook realtime processing scheme components. Figure 5 summarizes which alternatives were chosen yesteryear a multifariousness of realtime systems, both at Facebook in addition to inward the related literature.
Lessons learned
The newspaper includes a cracking lessons learned section. It says: "It is non plenty to render a framework for users to write applications. Ease of usage encompasses debugging, deployment, in addition to monitoring, as well. The value of tools that brand performance easier is underestimated. In our experience, every fourth dimension nosotros add together a novel tool, nosotros are surprised that nosotros managed without it."The highlights from this department are as follows:
- There is no unmarried linguistic communication that fits all usage cases. Needing dissimilar languages (and the dissimilar levels of ease of usage in addition to performance they provide) is the primary argue why Facebook has 3 dissimilar current processing systems, Puma, Swift, in addition to Stylus.
- The ease or hassle of deploying in addition to maintaining the application is every bit important. Making Puma deployment self-service allow them scale to the hundreds of information pipelines that usage Puma. (See Facebook's holistic configuration management most what type of systems Facebook employs to manage/facilitate deployments.)
- Once an app is deployed, nosotros require to monitor it: Is it using the right amount of parallelism? With Scribe, changing the parallelism is ofttimes merely changing the number of Scribe buckets in addition to restarting the nodes that output in addition to swallow that Scribe category. To reveal out the right amount of parallelism needed, Facebook uses alerts to reveal when an app is processing its Scribe input to a greater extent than tardily than the input is existence generated.
- Streaming versus batch processing is non an either/or decision. Originally, all information warehouse processing at Facebook was batch processing. Using a mix of streaming in addition to batch processing tin speed upwards long pipelines yesteryear hours.
Related posts
Facebook's software architectureHolistic Configuration Management at Facebook
Facebook's Mystery Machine: End-to-end Performance Analysis of Large-scale Internet Services
Measuring in addition to Understanding Consistency at Facebook
0 Response to "Realtime Information Processing At Facebook"
Post a Comment