Data Engineer
NTT DATA Global Delivery Services LtdJob Description
Data Engineer
"Position Overview
We are seeking an experienced Databricks Data Engineer to design and enhance enterprise document-processing and data-extraction solutions. The role will focus on ingesting structured and unstructured content, extracting business and policy information using LLM-based approaches, improving extraction accuracy, and making validated information available to downstream applications through an ontology-driven data layer.
The engineer will join a small, hands-on team working on document intelligence solutions supporting brokerage and underwriting workflows. A major focus will be redesigning existing pipelines to improve accuracy, performance, scalability, and maintainability while supporting a broader migration from Microsoft Fabric to Databricks.
Key Responsibilities
• Design, build, and optimize Databricks data pipelines for document ingestion, classification, extraction, validation, and downstream consumption.
• Process information from multiple document and communication formats, including:
o PDF documents
o Emails and attachments
o Excel/XLSX files
o Insurance applications
o Statements of Values (SOVs)
o Other structured and unstructured documents
• Improve existing extraction pipelines where LLMs produce false positives, incorrect metadata, or inaccurate field values.
• Develop validation and quality-control approaches that improve the reliability of extracted information.
• Implement document classification and content-aware extraction strategies.
• Handle source-specific extraction rules, such as extracting particular attributes from an email rather than from attached documents.
• Design pipelines that transform extracted information into consistent, structured JSON outputs for downstream applications.
• Contribute to the design and implementation of a custom ontology layer that represents key insurance business entities and relationships.
• Map extracted information into the ontology so applications can consume standardized business concepts rather than raw document fields.
• Address performance challenges associated with processing large or complex Excel/XLSX and SOV files.
• Optimize pipeline scalability and concurrency for production workloads.
• Work with PostgreSQL/Lakebase-based persistence for final structured outputs.
• Collaborate closely with brokerage, underwriting, AI, and data engineering teams to connect extracted application data with downstream underwriting processes.
Required Qualifications
• Strong hands-on experience with Databricks and production data engineering.
• Strong proficiency in Python, PySpark, and SQL.
• Experience building scalable ingestion and transformation pipelines on the Databricks platform.
• Experience processing unstructured and semi-structured data, particularly PDFs, emails, Excel files, JSON, and business documents.
• Experience integrating LLMs or AI models into data-processing pipelines.
• Understanding of prompt-based and structured data extraction from documents.
• Experience implementing data validation, reconciliation, or quality controls for AI-generated outputs.
• Strong understanding of data modeling, metadata, schemas, entity relationships, and semantic/ontology concepts.
• Experience designing APIs, JSON structures, or data products for downstream application consumption.
• Experience troubleshooting performance, scalability, and concurrency issues in production data pipelines.
• Experience working with cloud-based data architectures.
Minimum 7+ years of experience in data engineering, software development, or production data pipeline development.
Minimum 4+ years of hands-on experience with Databricks and scalable data-processing pipelines."
Experience Level
Senior LevelJob role
Job requirements
About company
Similar jobs you can apply for
Software / Web Developer
Application Engineer
Tridac Solution
QA / QC Executive
Pranavam Solutions Private Limited
Software Developer
Global Finacare
QA / QC Manager
SLV ServicesSoftware Developer
Toodles Fun And Learning CentreQuality Engineer
Shri Vinayaka Manpower RecruitmentsYou can expect a minimum salary of 0 INR. The salary offered will depend on your skills, experience and performance in the interview.
The candidate should have completed the required education and people who have 7 to 31 years are eligible to apply for this job. You can apply for more jobs in Bengaluru/Bangalore to get hired quickly.
The candidate should have sound communication skills and sound communication skills for this job.
Both Male and Female candidates can apply for this job.
No, it's not a work from home job and can't be done online. You can explore and apply for other work from home jobs in Bengaluru/Bangalore at apna.
No work-related deposit needs to be made during your employment with the company.
Go to the apna app and apply for this job. Click on the apply button and call HR directly to schedule your interview.
The last date to apply for this job is . For more details, download apna app and find Full Time jobs in Bengaluru/Bangalore . Through apna, you can find jobs in 64 cities across India. Join NOW!