Send me more jobs like this

Sorry, This job is expired.

Keywords / Skills : Python, Spark, UNIX, CRON, ETL, SQL

10 - 12 years
Posted: 2018-02-15

Industry
IT/ Computers - Software
Function
IT
Role
Software Engineer/ Programmer
Posted On
15th Feb 2018
Job Description
Python Developer (with Spark)

Job Location: Sunnyvale, CA
Duration: 6+ months contract

Job Description:
• Minimum 9+ years of software programming experience
• Strong experience in ETL
• SQL scripting experience
• Strong Python development experience around data management.
• Strong in Unix cron not just exposure
• Must have the ability to work in a dynamic, fast-paced environment
• Strong communication skills to interact with Agile team members

Data Sources: Cloud based as well as on premise example: SAP CRM, ECC, people soft, PLM, Calliduscloud

High-level architecture:
• Amazon S3 bucket have folders for specific source files delivered by Fusion middleware. Data quality check is not handled by the sources.
• Anaconda Python based programs are in place to clean any abnormalities in the source files and re-write them back in their respective folders this is to avoid load failures.
• most of the known errors are handled using this mechanism, moving forward you will require to enhance these scripts to include a fix for any new errors.
• currently mechanism is processing one file at a time, target is to move it to parallel processing.
• Next is a Python program for data pipeline, based on AWS API and snowflake API. This is to write files on snowflake EDW
• There are 3 different layers on the snowflake side Pre=processing, staging and base tables
Unix cron is used for scheduling purpose



About Company

Our Client is a Top 10 Business House with business activities ranging from financial services, information technology, power, BPO, chemicals etc.

About us:

T&M Services Consulting Pvt Ltd (T&M), is a professionally managed customer oriented HR Outsourcing & RPO company specilizing in Permanent Staffing, Temp Staffing, Executive Search for Middle and Senior Management, RPO and Employee Leasing and Training Service provider for leading companies in India and Abroad.

T&M has its HO at Mumbai, with presence in Delhi, Bangalore, Mumbai, Chennai and Hyderabad and is currently supporting projects and professionals in more than 400 locations.


Walkin for you