EP 50: Evolution of Enterprise Data Engineering in Gen AI Era

Shashank Akinapalli is a Senior Data Engineer at TCS where he leads cloud data lakehouse initiatives and real-time streaming architectures for supply chain intelligence. With 14 years of experience across IBM DataStage, Databricks, Snowflake, dbt, AWS, and Spark-based ecosystems, he has previously worked with USAA in banking and Blue Cross Blue Shield in healthcare.

IN THIS EPISODE:

▪ The biggest architectural shift when migrating from IBM DataStage to a modern cloud lakehouse — and why the technical migration is the easier half, while reconstructing encoded business knowledge is the real challenge

▪ How to choose between Snowflake and Databricks: Databricks for heavy transformation and Python workloads, Snowflake for structured SQL and BI, and where dbt fits as the layer that eliminates repetitive SQL code

▪ The real UNFI example: how batch processing caused inventory misrepresentation (800 units sold, system still showing 1,000 available), and how real-time Databricks pipelines eliminated that problem and unlocked fraud detection on top

▪ Data governance across three regulated domains: the four pillars — data quality, redundancy, least-privilege access, and lineage — and the three questions every framework must answer first

▪ Why CI/CD transformed data engineering: from manual, multi-team deployments with risky rollbacks to automated quality checks, Git version control, and the specific silent failure that finally convinces a team to invest

▪ How the data engineer role shifts over the next three years: from pipeline development to data product engineering, from reactive operations to AI-predicted failure prevention, and from human dashboards to AI agents making autonomous predictions

▪ Practical advice for data engineers who want to stay relevant: Python, PySpark, cloud architecture, and understanding the full data lifecycle from ingestion to consumption

FIND SHASHANK:

LinkedIn: https://www.linkedin.com/in/shashank-a-2aa629119/

Twitter / Instagram: @shashi001git