Senior Data Engineer - Teradata/Databricks

Work Locations : Any NTT locationExperience (Relevant) : 10+ YearsShift Timings : 2pm to 11pm ISTHybrid : 3 days/ weekStart date : ASAPPosition Overview :NTT DATA Americas is seeking a highly skilled Senior Data Engineer to support a Teradata Utilization Analysis engagement, which focuses on analyzing Teradata platform utilization to identify cost-reduction opportunities and deliver a repeatable, log-driven analysis solution.The ideal candidate combines deep Teradata DBA expertise with hands-on Databricks engineering capability and applied AI/ML skills. The analysis will be performed primarily in Databricks, leveraging DBQL logs, system metadata, and external scheduling/orchestration data sources.Key Responsibilities:- Ingest and validate 18+ months of Teradata DBQL logs including SQL text, object usage, timestamps, user/application IDs, row counts, and steps.- Integrate metadata from Autosys (scheduling), DataStage (orchestration), and MagicWand (observability) to supplement DBQL analysis.- Build end-to-end, log-driven analysis pipelines in Databricks to identify unused datasets, read-only (non-updating) datasets, and unused partitions within active datasets.- Capture and analyze CPU/IO resource usage and workload statistics (ResUsage) to quantify cost-reduction opportunities.- Classify data into cold, warm, and hot tiers; generate heatmaps of date/partition access patterns.- Develop a prioritized recommendation backlog with expected savings, risk levels, and required changes.- Apply AI/ML models or LLM-assisted analysis to detect access pattern anomalies, predict cold data candidates, and automate classification.- Produce and present deliverables: Observation Report, Workshop Notes & Action Log, and Final Readout for customer stakeholders.Required Skills & Experience :Teradata DBA Skills :- Teradata DBA / Administration : Primary platform under analysis; deep knowledge of system internals required.- Teradata DBQL (Database Query Logging) : Core data source - SQL text, object usage, timestamps, user/app IDs, row counts, steps.- Teradata System Views & Space Metadata : 5+ years - Object inventory, space analysis, last read/write tracking.- Teradata SQL & Performance Tuning : Writing complex analytical queries across DBQL and system catalog tables.- Teradata ResUsage / Workload Statistics : Quantifying CPU/IO cost impact for dataset removal/archival recommendations.- Data Classification (hot/warm/cold) : 3+ years - Categorizing datasets for tiered storage management recommendations.- ETL Pipeline Assessment (DataStage, Autosys) : 4+ years - Identifying and recommending decommission of stale ingestion pipelines.- Teradata BTEQ / FastExport / TPT : 3+ years - Data extraction, log export, and metadata collection utilities.Databricks Engineering Skills :- Databricks Platform (Workspaces, Clusters, Jobs) : Core analysis and solution delivery platform for the engagement.- Apache Spark (PySpark / Spark SQL) : Large-scale log ingestion, transformation, and analytical processing.- Delta Lake : Building reliable, ACID-compliant data pipelines for log-driven analysis.- Databricks Notebooks & Workflows : Developing repeatable, shareable analytical notebooks for customer handoff.- Python (pandas, numpy, matplotlib) : Data wrangling, statistical analysis, and visualization of usage heatmaps.- SQL (Spark SQL / ANSI SQL) : Analytical querying across large DBQL datasets in Databricks.- Unity Catalog / Data Governance in Databricks : Organizing deliverable assets and maintaining data lineage within the solution.- Dashboarding (Databricks SQL / Power BI) : Usage heatmaps, partition access patterns, and recommendation visualizations.AI / ML Skills :- Machine Learning (scikit-learn, MLflow) : Anomaly detection on access patterns, predictive cold-data classification.- LLM / Generative AI Integration : LLM-assisted log interpretation, automated recommendation narrative generation.- Feature Engineering on Time-Series Log Data : 3+ years - Extracting access frequency, recency, and seasonality signals from DBQL logs.- Clustering & Classification Algorithms : 3+ years - Unsupervised grouping of datasets by usage behavior for cold/warm/hot tiers.- MLflow / Experiment Tracking : 2+ years - Tracking model runs for reproducible analysis and customer handoff artifacts.- Natural Language Processing (NLP) : Parsing and classifying SQL text from DBQL for workload pattern analysis.- Data Visualization (Plotly, Matplotlib, Seaborn) : Generating usage heatmaps and partition access charts for stakeholder readouts.Preferred Qualifications :- Familiarity with DataStage orchestration log parsing.- Prior work on data platform cost optimization or cloud migration readiness assessments.- Experience working in healthcare payer data environments (HIPAA awareness).- Strong written communication skills for advisory deliverable authoring (Observation Reports, recommendation backlogs). (ref:hirist.tech)