One of the main pitfalls that we find when we go to the Cloud is to define the right data architecture that allows us to respond to current data needs and that, at the same time, gives us a sufficiently flexible path to face the changes that we will surely require. And all this with the sufficient guarantee that we will not have to redo all or a large part of the development done.
Here we must answer all these big questions:
At DataSpurs we have been advising companies for many years to define the best data architecture to meet each need. We are aware that this process is incremental and requires prior analysis, understanding the corporate strategy and applying the required technology in each use case.
Batch data integration is not the same as real-time integration through Kafka queues, for example, or the capture of changes from a relational database or the publication of REST endpoints in an API Gateway. You have to understand the need and implement the best solution.
This process must always take into account a complete common vision, that the tools allow flexibility, that we do not depend on multiple programming languages to make maintenance simple, that we can implement general privacy policies, end-to-end data governance processes, etc. This corporate vision will give us important competitive advantages over those who see each use case as independent.