Job Title:
Real-Time Stream Processing Engineer
About the Role:
We are a team of big thinkers who love to push boundaries and create new solutions. Together, we will build tomorrow's bank today, using world-leading technology and innovation.
Key Responsibilities:
* Design, develop, and optimize Flink data streams to implement event buffering strategies based on fixed time intervals, sliding windows, session windows, and custom triggers.
* Develop and maintain robust Flink pipelines for processing and aggregating data from various real-time data sources, including Kafka, Amazon MSK, etc.
* Implement and manage stateful stream processing with checkpointing, fault tolerance, and exactly-once semantics.
* Create custom Flink Process Functions with precise time-based buffering logic and timers for controlling event triggers and processing.
* Collaborate with data engineers and devops to ensure seamless integration of Flink applications with downstream data storage and processing systems (e.g., S3, databases, data lakes).
* Monitor and fine-tune Flink job performance using built-in metrics, profiling tools, and customized metrics and alerts to ensure minimal event lag, low latency, and high throughput.
* Identify and resolve bottlenecks, data loss issues, and challenges related to buffering, windowing, and high-volume stream processing.
* Implement and configure event retention policies on streaming sources such as Apache Kafka and ensure end-to-end data consistency and accuracy.
Requirements:
* Expert-level knowledge of Apache Flink and its windowing mechanisms (Tumbling, Sliding, Session windows, Global windows, etc.).
* Hands-on experience with data streaming platforms like Apache Kafka, Amazon MSK, and related technologies.
* Strong proficiency in Python for Flink application development.
* Experience with monitoring, logging, and alerting in distributed data processing environments (e.g., Grafana, Prometheus, CloudWatch).
Preferred Qualifications:
* Certification with AWS Data Engineering and AWS Solution Architect Professional (e.g., AWS Kinesis Data Analytics, AWS Lambda).
* Experience with NoSQL or SQL databases used in conjunction with Flink.
* Familiarity with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or CDK.
About Us:
At {company}, we're committed to making a positive impact on our customers' lives. We're passionate about innovation, collaboration, and continuous improvement. If you're looking for a challenging and rewarding role that aligns with your values and career goals, join us!