Apache Beam is an abstraction layer for stream processing systems like Apache Flink, Apache Spark (streaming), Apache Apex, and Apache Storm.

Beam Developers CWiki:

• https://cwiki.apache.org/confluence/display/BEAM/Apache+Beam

Introduction videos:

• Big Data Processing with Apache Beam Python | SciPy 2017 | Robert Bradshaw - explains the story behind Beam
• Streaming data processing pipelines with Apache Beam - PyCon APAC 2018

So far Beam is nothing more than a troublemaker:

• As of Sep 2021 Apache Beam Python library is not compatible with Python 3.9: The Python SDK supports Python 3.6, 3.7, and 3.8.
• its portable runnners (e.g. SparkRunner) are not handling streaming
• it seems that Python Beam + Flink + Kafka can only work in VMs (not in Kubernetes)
• BEAM's Jira Kanban shows quite a few Beam+Kafka related issues

Apache Beam Alternatives & Comparisons

• https://stackshare.io/apache-beam

• Apache Beam KafkaIO Example