Shivam Rajput

Engineered and maintained robust ETL pipelines utilizing AWS services to ensure seamless data ingestion, transformation, and loading processes.

• Developed a Python script leveraging boto3 and concurrent.futures to optimize data migration between Amazon S3 buckets, achieving a 30% reduction in transfer time.
• Created and optimized stored procedures in Amazon Redshift for complex data operations, significantly enhancing performance and scalability.
• Maintained robust ETL pipelines using AWS Glue, Amazon Redshift, and Amazon S3, ensuring seamless data ingestion, transformation, and loading processes.

Currently leads the development and optimization of robust data pipelines and data lake architecture for scalable and performant analytics.

• Built and scheduled data pipelines using Apache Airflow to ingest data from Google Sheets, REST APIs, and MongoDB into Trino tables, ensuring reliable data availability.
• Implemented Debezium and Kafka for real-time change data capture from MongoDB collections, centralizing data into the core platform.
• Contributed to an in-house data architecture leveraging Apache Iceberg, Amazon S3, and Trino, optimizing query performance for scalable and performant analytics.
• Supported data transformation workflows using Apache Spark for efficient batch processing within a distributed data lake environment.

Developed an AI-powered staffing recommendation system leveraging NLP and embedding-based similarity search, securing 2nd place in the Gen AI Hackathon.

Developed a microservices-based system for product management and user interactions, focusing on efficient, decoupled services.

Awarded for developing an innovative AI-powered staffing recommendation system that leveraged NLP and embedding-based similarity search.

Achieved Codeforces Expert status with a max rating of 1727 and a Global Rank of 675 in Codeforces Round 927, solving over 350+ Data Structures & Algorithms problems, showcasing advanced problem-solving and algorithmic skills.

Achieved a top 0.5% ranking among over 1 Million candidates in the highly competitive JEE ADVANCED 2020 examination, demonstrating exceptional aptitude in science and engineering.