is "distributed event streaming platform"
was created by LinkedIn
Use-cases from conductor site
- Messaging systems
- Activity Tracking
- Gather metrics from many different locations, for example, IoT devices
- Application logs analysis
- De-coupling of system dependencies
- Integration with Big Data technologies like Spark, Flink, Storm, Hadoop
- Event-sourcing store You can find a list of use cases at https://kafka.apache.org/uses
mental model
cluster > broker > topic > partition :: consumer group > consumer
acts as buffer. difference from queues
tidbits -> kafka is used by seven out of ten banks in their technical systems.
explore
when is doubt check the Kafka UI by conductor which is more like an interface for command line with respect to all critical commands and parameters. they also use same defaults
when i talk to many full stack developers i see that they can draw the similarities between kafka and redux
tips
- better to use xfs instead of ext4
- number of consumers is limited by the number of partitions. so the limit for parallelisation is number of partitions
- increasing partitions is easy. decreasing is difficult
- get better intuition around this
kafka metrics
https://docs.confluent.io/platform/current/kafka/monitoring.html#localtimems
Referenced in:
All notes