Send me more jobs like this

Keywords / Skills : Python, Spark, UNIX, CRON, ETL, SQL

10 - 12 years
Posted: 2018-02-15

IT/ Computers - Software
Software Engineer/ Programmer
Posted On
15th Feb 2018
Job Description
Python Developer (with Spark)

Job Location: Sunnyvale, CA
Duration: 6+ months contract

Job Description:
• Minimum 9+ years of software programming experience
• Strong experience in ETL
• SQL scripting experience
• Strong Python development experience around data management.
• Strong in Unix cron not just exposure
• Must have the ability to work in a dynamic, fast-paced environment
• Strong communication skills to interact with Agile team members

Data Sources: Cloud based as well as on premise example: SAP CRM, ECC, people soft, PLM, Calliduscloud

High-level architecture:
• Amazon S3 bucket have folders for specific source files delivered by Fusion middleware. Data quality check is not handled by the sources.
• Anaconda Python based programs are in place to clean any abnormalities in the source files and re-write them back in their respective folders this is to avoid load failures.
• most of the known errors are handled using this mechanism, moving forward you will require to enhance these scripts to include a fix for any new errors.
• currently mechanism is processing one file at a time, target is to move it to parallel processing.
• Next is a Python program for data pipeline, based on AWS API and snowflake API. This is to write files on snowflake EDW
• There are 3 different layers on the snowflake side Pre=processing, staging and base tables
Unix cron is used for scheduling purpose

About Company

Our Client is a Top 10 Business House with business activities ranging from financial services, information technology, power, BPO, chemicals etc.

About us:

T&M Services Consulting Pvt Ltd (T&M), is a professionally managed customer oriented HR Outsourcing & RPO company specilizing in Permanent Staffing, Temp Staffing, Executive Search for Middle and Senior Management, RPO and Employee Leasing and Training Service provider for leading companies in India and Abroad.

T&M has its HO at Mumbai, with presence in Delhi, Bangalore, Mumbai, Chennai and Hyderabad and is currently supporting projects and professionals in more than 400 locations.
Similar Jobs
View All Similar Jobs

Walkin for you