Taming the Wild West of Background Jobs: A Tale of Three Bug Fixes in Landing

In the world of background jobs, things can get a little hairy. Unexpected data, deleted resources, and security audits can all throw a wrench in the gears. This is the story of how we tackled three such gremlins in our Landing project, a service designed to manage and publish developer logs.

The Scenario

Our team was alerted to a few issues plaguing the background processing of posts. These ranged from projects slipping through filters to scheduled posts vanishing into thin air and security audits raising false alarms.

The First Catch: Filtering Disabled Projects

The first bug involved our system inadvertently processing commits even when their associated projects were disabled. This meant that commits lacking a resolved project ID were not being excluded from the AutoSyncPostGenerationJob.

To remedy this, we added a whereNotNull('project_id') clause to the job's filters. This ensures that only commits with a valid project ID are processed, effectively respecting the disabled project configurations. It's like adding a gatekeeper to the processing pipeline, ensuring only the right data gets through.

The Second Catch: Vanishing Scheduled Posts

Next, we faced an issue where scheduled posts would sometimes disappear before their scheduled publishing time. This occurred when a post was deleted but the associated scheduled jobs were still lingering, attempting to publish a non-existent post.

Our fix involved two key steps:

1. Canceling any pending or processing ScheduledPosts when a Post is deleted. This is done during the deleting event of a Post.
2. Verifying the post's existence within ProcessScheduledPosts before dispatching any publish jobs. This acts as a final check to ensure we're not attempting to publish a ghost post.

This is akin to ensuring the delivery address exists before sending a package – preventing wasted effort and potential errors.

The Third Catch: False Security Alarms

Finally, our security audits were flagging numerous false positives due to placeholder credentials, environment variable examples, and script/command names mentioned in blog posts. While these mentions were harmless, they cluttered the audit logs and made it harder to identify genuine security concerns.

To address this, we implemented more precise filtering rules within our security audit tooling. This allows us to ignore these benign mentions, reducing noise and allowing us to focus on legitimate security threats.

The Lesson

Taming background jobs requires vigilance and a proactive approach. By implementing robust filtering, thorough resource management, and fine-tuning our security tooling, we've made our Landing project a more reliable and secure platform for developer logs.