In software development, there is a dangerous trap: the belief that if a system is "working," it is perfect. But as a senior engineer, I have learned that the true art of our craft is not just making things work — it is knowing when to simplify them, even when the "legacy" is protected by management.
The sacred legacy: the "if it ain't broke" barrier
One of the toughest hurdles in any tech company is dealing with code written by a predecessor. When a system — no matter how over-engineered — is currently functional, it becomes "sacred."
In my experience, I encountered a workflow that was built using a complex GenStage pipeline. To management, this was a high-end, enterprise solution. Because it was working, there was immediate resistance to any change. They did not see the technical debt; they only saw a running system. This is where the battle of an engineer begins: justifying a move to simplicity when the complex version is already in production.
The technical trap: GenStage vs. GenServer
The specific case involved fetching JPEG snapshots from CCTV cameras at set intervals, broadcasting them via Phoenix Sockets, and uploading them to the cloud.
The previous architecture used GenStage. While GenStage is a powerful tool for high-volume data streams with back-pressure, using it for periodic polling was a classic case of over-engineering.
1. The binary memory bloat
The biggest issue was memory. In Elixir, large binaries (like JPEGs) are reference-counted. Because GenStage uses internal buffers to manage flow between stages, it was holding onto these image references much longer than necessary. Every time we added a new feature or a new stage, binary memory usage skyrocketed. We were not just processing images; we were leaking memory because the pipeline refused to let go of the data until every stage had demanded it.
2. The simplicity of GenServer
For this task, a standard GenServer was the logical choice. A GenServer is lightweight, native, and perfectly capable of handling a fetch-broadcast-upload cycle. Once the task is done, the memory is cleared. No buffers, no complex demand-driven logic — just clean, predictable performance.
The barrier: management, complexity, and obligation
The real challenge was not only the refactor — it was persuading leadership. When management is not deeply technical, it is easy for a quick web search to mistake complexity for "robustness." I heard arguments like: GenStage is for high-performance pipelines, so we should keep it — without the nuance of binary heap behavior or when back-pressure actually matters.
Handling non-technical management in those moments is genuinely difficult. And once you can see the problem clearly, you feel obliged to solve it — even when the organizational default is to leave the working system untouched. I had to build prototypes and proof-of-concepts to show that our sophisticated setup was a bottleneck that slowed us down and consumed RAM.
Conclusion: engineering is persuasion
This experience taught me that being a senior developer is 50% technical expertise and 50% communication. You have to convince management that:
- Simplification is not a downgrade; it is an optimization.
- Over-engineering is a silent killer of scalability.
- A "working" system can still be a broken architecture.