As year changed from 2019 to 2020, people in social media posted a lot about how they have spent their 2010s. I will take a little different aspect while looking into past decade and present you my three biggest architectural failures.
Interop
This story begins at the beginning of 2010s. I was just promoted to Software Architect and got an invite into the Architect Board. Architect Board was basically a group of people who were responsible for their business units software architectural decisions. In board meetings we discussed about upcoming changes and what technologies we should embrace. In these meetings, we invented the Interop.
Our main products were monolith C# applications that were branched from same source (for historical reasons) and then customized to do their product special stuff really well. So all the products had common ancestor, but they also contained lot of own code. At some point we realized that we need to share more code between these products, so we created a concept called Modules.
Modules were application parts that didn’t contain any customization for products, so they were usable as they are in multiple products. To implement these Modules, we need to solve one problem: How modules can interact with underlying products to get some basic data like customer or order? We didn’t have proper microservice architecture at that point, so we had to share database through shared data classes (product customer and module customer was same class). Data classes was implemented inside the monolith application, but we could not add reference into monolith application from modules, because modules were interacting with multiple different product. They could not just reference one product and get data classes.
In one blooming day we got a brilliant idea in Architect Board. What if we create a shared library between modules and the monolith application and add all the data classes into it. All the products and modules can reference this shared library. We just need to place it into common place (with Git submodules or something like that)
So we ended up into solution like this:
It took about three months for us to realize, that we had created something monstrous that cannot be killed easily….
The Problem
At first everything went nicely and we were happy with the Interop. It was like new romance. You see some faults in it, but because you are so blindly in love, you don’t care about them that much, until it hits you right into face.
One day one of our developer wanted to change interop data class. He did the change, implemented logic inside module and pushed new version out of the module. Of course this change also meant, that we had new version of Interop available for products. When products are ready to update the module, they would apply data class change into their databases. Few of our products did not care much about this change, so they did not update the module or Interop. Few weeks later one of those products required changes into another module, but the Interop contained changes that was not applied yet to product, so they could not just add new fields into Interop. They could have branched the Interop from previous version, add changes to there, and then implement them also into master, or implement the missing Interop changes and then implement new changes into Interop. However few of the products were not updated that often, which eventually meant that when they wanted to have new version of one module, it required them to implement all the other changes that was introduced into Interop, even for those modules that were not used in that product.
This lead into problem that we could not simply take new version of one module and use it in product. We had to always implement all the Interop changes. Now when you scale up the module development, so that Interop is updated two, maybe three times a day. Think about what work it requires to update one product that has not update the Interop in one or two months.
Solving the problem
Well this story has also the happy ending, don’t worry. When this issue caused too much pain in development, we knew that we had to split the Interop. We created own Interops for each of the modules so that modules could be updated independently and unused modules did not require changes into product. However we could not just remove the original Interop, because it would meant that all the products and modules would stop working. We had to move the data classes from Interop one by one after creating new “Module Interops” and fix all the product references. We had also implemented dependencies inside the Interop between modules without actually noticing it, because we did not have any restrictions inside the Interop. It took almost three years to get rid of the Interop for good.
The lesson of the story
When working with modules (like microservices), don’t create single communication points that are used in all the modules or microservices. Separate dependencies into module based libraries and if possible, don’t even share the database because it is also single communication point like data classes.