bento

Introduction to Bento

Bento is a cutting-edge stream processing framework designed to seamlessly handle data from various sources and direct it to different destinations, known as sinks. It is robust and performs tasks like hydration, enrichment, transformation, and filtering of data payloads. Bento stands out with its powerful mapping language, making it versatile and adaptable. It can be effortlessly deployed as a static binary, Docker image, or even as a serverless function, making it incredibly cloud-native and easy to integrate into existing pipelines.

Features and Capabilities

Declarative Configuration

Bento is highly declarative, allowing users to set up data pipelines with a simple configuration file. This file specifies the inputs, the desired data processing stages, and the outputs. Here's an example:

input:
  gcp_pubsub:
    project: foo
    subscription: bar

pipeline:
  processors:
    - mapping: |
        root.message = this
        root.meta.link_count = this.links.length()
        root.user.age = this.user.age.number()

output:
  redis_streams:
    url: tcp://TODO:6379
    stream: baz
    max_in_flight: 20

Delivery Guarantees

Bento ensures at-least-once delivery of messages even in challenging conditions like server crashes or disk corruption. This reliability is achieved with an in-process transaction model that doesn't require disk-persisted state, making deployment and scaling a breeze.

Supported Sources and Sinks

Bento supports an extensive range of sources and sinks, including major cloud services like AWS, Azure, and GCP, as well as technologies such as Kafka, NATS, MQTT, and Redis. The system is constantly evolving, adding new connectors as required. If additional functionalities are needed, users can suggest improvements or new features.

How to Get Started

Installation

To run Bento, users can download the Docker image or compile it from the source:

docker pull ghcr.io/warpstreamlabs/bento

For further setup details, users should refer to the getting started guide.

Running Bento

Once installed, Bento can be executed with a provided configuration file:

bento -c ./config.yaml

Alternatively, it can be run using Docker, either with a configuration file or by specifying configurations using flags.

Monitoring and Observability

Bento supports health checks and metric exposure for effective monitoring:

• Health Checks: Bento provides endpoints for liveness (/ping) and readiness (/ready) to ensure the application is functioning correctly.
• Metrics: It can export metrics to systems like StatsD or Prometheus, ensuring users can track performance and operational statistics.
• Tracing: Open telemetry tracing events are emitted to visualize data processing through the pipeline.

Customization and Enhancement

Bento is customizable with plugins, and users can develop their own using Go programming language. External components can also be included by building Bento locally with specific build tags.

Contributions

Bento is open to contributions. Developers are encouraged to engage with the community, contribute code, or suggest improvements by following the project’s contribution guidelines.

Conclusion

Bento is an efficient, adaptable, and reliable stream processor, suitable for a range of data-driven applications. Its comprehensive feature set and flexibility make it a powerful tool for modern data engineering tasks. Whether integrating with popular cloud services or extending its capabilities through custom plugins, Bento is designed to handle the demands of real-world data processing at scale.