Engineered a real-time stock market data pipeline using Python, Kafka, and AWS, streaming live tick data from producers to consumers with under 500 ms end-to-end latency.
Scaled event processing with partitioned Kafka topics on AWS EC2, enabling 10,000+ messages per second and improving throughput during bursty market activity.
Automated durable storage and metadata management by writing JSON records to S3, then using AWS Glue Crawler + Glue Catalog to reduce manual schema work by 80%.
Enabled serverless analytics with Amazon Athena, turning raw streaming data into queryable insights in minutes instead of hours while reducing infrastructure costs by up to 40%.
Made with Bullet