Material Conditions in Software Development
More often than not, material conditions determine our life, they determine the amount of “liberty” we have and the opportunities we can take. Our perception of life and the world around us, depends exclusively in the context we grew in, in the material conditions our life developed with.
This is how life works, I do not think that truth and reality are concepts that apply the same way for all of us. Like the observer effect in quantum mechanics, my reality is different from yours, not because the universe decides its presentation according to who is experiencing it, but because my perspective about it is unique and can not be replicated.
This, I guess, applies for software engineering and systems design too.
How many times, as a junior engineer, you had been tempted, by your lack of understanding, to rewrite an entire code base with the naive excuse of “How is this possible? This is a bad implementation, I can for sure do it better”. I have fallen into this trap many times in the past, but it is important to recognize that this attitude towards solving problems is not sustainable.
Understanding that material conditions determine which is the correct design choice at a given time will free us to take action into whatever solutions seems to fit the problem at the time, not caring too much about the solutions that seem correct but end up being more complex and complicated.
Have you ever found yourself as a one man army trying to implement a hole solution by yourself just by playing and making proofs of concept? That was my situation some years ago trying to start a data lake initiative, after reading and trying to arrive to a conclusive architecture a specific question arouse. Was I going to implement an Apache Kafka cluster with its respective Debezium connectors cluster to make a real time data extraction or was I going to use a managed solution like AWS Database Migration Service?
I can confidently say that as a whole, Apache Kafka seems a better solution, I have been using DMS for some time now and I do have some disagreements with the service, I do not like how it works and I think that at least with the CDC functionality, DMS has room for improvement.
But then again, which was my situation at the time? I was the only one implementing what was going to be the “data architecture” in the organization, delivering some value as soon as posible to highlight the improvements of this solution was important and last but not least, I was limited in my knowledge doing cluster administration.
Looking back, implementing Kafka would have ease my current experience, but Kafka was not the correct decision back then, this, taking the material conditions into consideration. DMS was a great service to start generating value and building what now a days I can call the data team inside the org.
Conditions, like humans, change, and decisions made in the past become obsolete. Today, replacing DMS for Kafka seems inevitable, but arriving to that necessary change was a whole journey by itself. I am no longer alone nor I have to demonstrate the value in having a strong data architecture, the conditions have changed and the correct answer for that specific question then changed too.
Note: This is a personal blog, if some fact is wrong (and someday one will be) please be kind and let me know, this is not formal advice nor writing.