Data Engineer (#669938)

Will integrate Tableau Cloud and Tableau dashboards with internal VisOps platform to monitor real-time transactions, sales, inventory, and price lock Dashboards to support dynamic pricing decisions and operational visibility. Perform data migration initiatives, transferring large volumes of critical datasets from different POS systems (including MongoDB) to cloud-based environments (Azure Data Lake), supporting operational and analytical needs using Python Script. Build and monitor Databricks jobs for transforming raw fuel pricing data from multiple sources (including MongoDB and PostgreSQL) into structured, analytical views used in downstream reporting. Develop and optimize data pipelines using PySpark and SQL in Databricks to clean, aggregate, and enrich multiple datasets like pricing, transaction, inventory, also ensure data freshness by monitoring data lags and implementing auto-refresh logic. Create and optimize views in Databricks ahead of Tableau integration to enhance dashboard performance, reduce load times, and ensure efficient data retrieval for real-time analytics. Write PySpark code in Databricks for complex data transformations and analytics pipelines. Design and implement Tableau Prep flows and publish curated data sources to Tableau Cloud to streamline data processing and significantly improve dashboard performance and responsiveness. Implement data governance frameworks and access control policies in Tableau and Unity Catalog. Develop and manage ETL processes within the Data Lake using Databricks and PySpark to clean, transform, and structure data, and publish these optimized data sources to Tableau Cloud to enhance dashboard performance, ensuring data consistency, and support real-time analytics. Perform end-to-end troubleshooting across the Data Lake and Tableau environments, resolving data pipeline failures, query performance issues, and dashboard rendering problems to ensure seamless data availability and optimal user experience. Integrate and consume external and internal APIs within the Data Lake environment using PySpark and SQL to ingest real-time data streams. Leverage Kafka topics across multiple clusters in Databricks to enable scalable, distributed data ingestion pipelines, supporting use cases such as real-time fuel price updates, transactional logging, and event-driven analytics, while ensuring high availability and fault tolerance across the data infrastructure. Create automated alerts in the Data Lake using Databricks by monitoring key tables and views to validate data accuracy, freshness, and completeness, ensuring timely identification of anomalies or delays in upstream data pipelines. Utilize MongoDB’s NoSQL to perform document-based data analysis, execute flexible search queries, and troubleshoot data integrity and availability issues within semi-structured datasets.

Position requires up to 100% domestic travel. This position is for full-time, salaried (W-2), permanent employment.

REQUIREMENTS: Master’s Degree or foreign equivalent in Information Technology, Information Systems, Business Analytics, Engineering, or a related field.

Please reference Job Number 669938 when sending resumes. Please mail resumes to: HR, Beacon Hill Solutions Group, LLC, 20 Ashburton Place, 5^th Floor, Boston, MA 02108.

California residents: Qualified applications with arrest or conviction records will be considered for employment in accordance with the Los Angeles County Fair Chance Ordinance for Employers and the California Fair Chance Act.

If you would like to complete our voluntary self-identification form, please click here or copy and paste the following link into an open window in your browser: https://jobs.beaconhillstaffing.com/jobs/eeoc/

Completion of this form is voluntary and will not affect your opportunity for employment, or the terms or conditions of your employment. This form will be used for reporting purposes only and will be kept separate from all other records.

Beacon Hill offers a robust benefit package including, but not limited to, medical, dental, vision, and federal and state leave programs as required by applicable agency regulations to those that meet eligibility.  Upon successfully being hired, details will be provided related to our benefit offerings.