Principal Golang Developer - Workflow Automation Specialist

About Lumen

Lumen connects the world. We are igniting business growth by connecting people, data and applications - quickly, securely, and effortlessly. Together, we are building a culture and company from the people up - committed to teamwork, trust and transparency. People power progress.

Lumen's commitment to workplace inclusion and employee support shines bright. We've made the Newsweek 2024 Greatest Workplaces for Diversity list and achieved a perfect score of 100 on the Human Rights Campaign Corporate Equality Index (CEI) for the fifth consecutive year. Plus, we're the top employer in the communications and telecom industry, ranking 12th overall across all industries in The American Opportunity Index.

We're looking for top-tier talent and offer the flexibility you need to thrive and deliver lasting impact. Join us as we digitally connect the world and shape the future.

The Role

We are seeking a Principal Golang Developer with advanced expertise in Temporal and workflow automation to build, optimize, and maintain resilient, scalable systems. This role requires deep knowledge of distributed systems, resilience patterns for messaging-based workflows, and the ability to design and implement reliable, efficient automation solutions. The ideal candidate is self-sufficient, technically proficient, and passionate about delivering seamless, high-performance orchestration systems.

Work Location

• This is an US work from home position.

The Main Responsibilities

Temporal Workflow Development:

• Design and develop complex workflows and activities using Temporal, leveraging features such as retry policies, saga patterns, child workflows, signals, and queries.

• Implement workflows with state persistence, ensuring recoverability, consistency, and reliability in distributed environments.

• Optimize workflows for performance, fault tolerance, and low-latency execution.

• Handle long-running processes effectively, managing timeouts, scheduled tasks, and compensations.

Resilience Patterns in Messaging-Based Systems:

• Architect workflows with robust resilience patterns, including retry with backoff, dead-letter queues, and idempotency handling.

• Ensure reliable message processing in distributed environments using tools like AWS SQS, SNS, EventBridge, or Kafka.

• Design for at-least-once and exactly-once processing semantics to ensure workflow consistency.

• Implement circuit breakers, load shedding, and backpressure handling to maintain system stability under load.

Distributed System Design:

• Develop scalable, fault-tolerant systems with a focus on high-throughput workflows and dynamic task scheduling.

• Design for efficient parallel task execution, fan-out/fan-in patterns, and event-driven architectures.

• Build systems that handle graceful degradation, ensuring minimal disruption during failures.

Monitoring, Observability, and Fallout Handling:

• Implement monitoring and alerting for workflows and systems using tools like Temporal Web UI, Prometheus, Grafana, and AWS CloudWatch.

• Set up observability for workflow state transitions, latency metrics, and error patterns to proactively identify and resolve issues.

• Design robust fallout management systems, including automated retries, compensating transactions, and manual intervention mechanisms when necessary.

Collaboration & Best Practices:

• Work with cross-functional teams to integrate workflows seamlessly into broader system architectures.

• Establish and promote best practices for workflow automation, resilience patterns, and distributed system performance.

• Mentor and guide team members in Temporal concepts, workflow patterns, and resilient design principles.

What We Look For in a Candidate

Must-Have Skills:

Temporal Expertise:

• Advanced experience with Temporal, including designing workflows, managing activities, and optimizing system performance.

• Strong understanding of Temporal features such as activity retries, signals, queries, child workflows, and saga patterns.

• Familiarity with Temporal's persistence and scalability mechanisms, ensuring efficient state management across systems.

Golang Proficiency:

• Advanced Golang development skills, with experience building high-performance, scalable systems.

• Proficiency in Temporal SDKs for Golang and implementing complex workflows in production environments.

Resilience Patterns & Distributed Systems:

• Strong knowledge of messaging-based systems and resilience techniques, such as idempotency, retry with exponential backoff, and dead-letter queues.

• Expertise in distributed systems design, including fault tolerance, data consistency models, and message passing.

Monitoring & Observability:

• Proficiency in using Prometheus, Grafana, and CloudWatch for distributed system monitoring and debugging.

• Experience implementing observability for Temporal workflows, including state tracking and error analysis.

Nice-to-Have Skills:

• Familiarity with containerization using Docker and orchestration with Kubernetes.

• Experience with AWS-native tools like Step Functions, DynamoDB, or Aurora.

• Exposure to chaos engineering techniques to test system resilience.

• Knowledge of event-driven systems and hybrid architectures involving Temporal and messaging tools.

Compensation

This information reflects the anticipated base salary range for this position based on current national data. Minimums and maximums may vary based on location. Individual pay is based on skills, experience and other relevant factors.

Location Based Pay Ranges

$149,084 - $198,779 in these states: AL AR AZ FL GA IA ID IN KS KY LA ME MO MS MT ND NE NM OH OK PA SC SD TN UT VT WI WV WY

$156,539 - $208,718 in these states: CO HI MI MN NC NH NV OR RI

$163,993 - $218,657 in these states: AK CA CT DC DE IL MA MD NJ NY TX VA WA

Lumen offers a comprehensive package featuring a broad range of Health, Life, Voluntary Lifestyle benefits and other perks that enhance your physical, mental, emotional and financial wellbeing. We're able to answer any additional questions you may have about our bonus structure (short-term incentives, long-term incentives and/or sales compensation) as you move through the selection process.

Lumen Benefits (~~~)

Bonus Structure

What to Expect Next

Requisition #: 336313

Background Screening

If you are selected for a position, there will be a background screen, which may include checks for criminal records and/or motor vehicle reports and/or drug screening, depending on the position requirements. For more information on these checks, please refer to the Post Offer section of our FAQ page (~~~) . Job-related concerns identified during the background screening may disqualify you from the new position or your current role. Background results will be evaluated on a case-by-case basis.

Pursuant to the San Francisco Fair Chance Ordinance, we will consider for employment qualified applicants with arrest and conviction records.

Equal Employment Opportunities

We are committed to providing equal employment opportunities to all persons regardless of race, color, ancestry, citizenship, national origin, religion, veteran status, disability, genetic characteristic or information, age, gender, sexual orientation, gender identity, gender expression, marital status, family status, pregnancy, or other legally protected status (collectively, "protected statuses"). We do not tolerate unlawful discrimination in any employment decisions, including recruiting, hiring, compensation, promotion, benefits, discipline, termination, job assignments or training.

Disclaimer

The job responsibilities described above indicate the general nature and level of work performed by employees within this classification. It is not intended to include a comprehensive inventory of all duties and responsibilities for this job. Job duties and responsibilities are subject to change based on evolving business needs and conditions.

In any materials you submit, you may redact or remove age-identifying information such as age, date of birth, or dates of school attendance or graduation. You will not be penalized for redacting or removing this information.

Please be advised that Lumen does not require any form of payment from job applicants during the recruitment process. All legitimate job openings will be posted on our official website or communicated through official company email addresses. If you encounter any job offers that request payment in exchange for employment at Lumen, they are not for employment with us, but may relate to another company with a similar name.

Application Deadline

01/02/2025