As a growing data consultancy boutique, we get more and more questions to review and architect data platforms. While growing, we are also maturing the architecture practice at Dataroots.

What is Architecture?

We can obviously not discuss architecture without some reflection on the term itself in the context of ICT solutions in general, and data platforms specifically. A topic which immediately proves to be not that easy to grasp. When brainstorming the subject, we easily talked about the responsibilities of an architect like understanding the customer needs, capturing requirements, translating those into a high-level design, etc. On the other hand, defining ‘architecture’ seemed very difficult.

IT architecture clearly is an abstract notion.

When turning to the internet to figure out what other and smarter people have to say on the subject, Martin Fowler is obviously a force to reckon with in the world of software. There is a whole section of his website dedicated to the subject: https://martinfowler.com/architecture. The available material clearly demonstrates the same struggle.

“Architecture is the highest-level concept of a system in its environment. The architecture of a system is its organisation or structure of significant components interacting through interfaces, those components being composed of successively smaller components and interfaces.”

'highest-level concept' lies obviously in the eye of the beholder, because different people have a different understanding of the same system. Think user vs developer. Martin Fowler apparently had a long online discussion about the topic with Ralph Johnson, professor at the University of Illinois, father of Smalltalk and member of the infamous Gang of Four. So, they iterated over a few definitions ...

Architecture is the shared understanding that the expert developers have of the system design

Architecture is about the decisions you wish you could get right early in a project

... to finally settle for

Architecture is about the important stuff ... whatever that is

Let that sink in for a minute... Although quite vague, it does contain depth. This definition covers the fact that architecture is a social construct, that it depends heavily on context and audience, and that is is about design.

Thinking about architecture is to decide what is important and then expend energy to keep those architectural elements in good condition. This notion of time is crucial. It implies evolution. What do we consider important? Which part should be able to absorb change easily? Software is by definition easy to change. However, every time you add a dimension that must be 'easy to change', the software construct becomes a bit more complex. And so Ralph Johnson correctly observes: "And making everything easy to change, makes the entire system very complex. Complexity is what makes software hard to change. That, and duplication." He concludes that "software is not limited by physics, like buildings are. It is limited by imagination, by design, by organization. In short, it is limited by properties of people, not by properties of the world. We have met the enemy, and he is us.”

What is an Architect?

If architecture and complexity of software is a social construct and not an exact science, then architecture must be a craft based on experience and practice. To become an architect you have to learn to understand requirements and needs, to distill what is important, where change needs to be absorbed easily … To be watchful for the invariants. This practice seems to be fundamental to all types of architects in IT, be it a solution architect, an application architect, a cloud architect or a data platform architect.

So, then what is an architect? Well, not this ...

One could argue that such work doesn't belong to the field of IT architecture, caters to a completely different kind of audience and is often practiced in the confinement of ivory towers. During a meeting with some customer, we even picked up the term 'Ayatollah architects'.

What we expect of our architects is to be pragmatic, to focus on the important things and for the intended audience. Our expert engineers in the team should feel that breath of fresh air.

An architect collaborates intensely with the team, captures requirements, translates the relevant info into a design, aligns that design with the target organisation, keeps evolution and change in mind while looking out for important issues and while mentoring the team.

The architect is he who focuses on the important stuff.

Tech Lead vs Architect

Then, "What is a tech lead?" I hear you say.

What is clear is that architecture is a set of responsibilities. On a single project. the tech lead can be the architect as well, while in a program of projects the architect can be a separate position. So, there is certainly a factor of scale involved.

What does architecture do for our teams?

First and foremost, the architectural responsibilities cover the quality of the design. What is important? to align requirements, needs and priorities with the design and technical roadmap. Again, while taking expected change into account.

Of course, the technical consistency of the proposed design must be guaranteed such that technical components can work together, technical standards are followed, and teams can be coached in technical excellence.

Along the way, decisions and architectural choices related to the project are documented and kept up to date.

Finally our engineering teams can ask for a review of their solution design to get advice on their solution design or get advice on potential improvements.

What does architecture do for our customers?

Depending on the way the customer wants to work, we consult to implement ad-hoc solutions on a lightweight template data platform (use case first) or design a complete tailor-made platform starting from a requirements assessment (platform first).

When an existing platform is already in place, an outside-in view on technical design and roadmap can be more appropriate.

Next steps

As we are just bootstrapping architecture as an official practice at Dataroots, we are just scratching the surface. There are many next steps to take. We’ll be digging deeper into those architectural responsibilities, but a first internal poll revealed some dire needs, so we are prioritising a few things.

Having a common visual language to describe architectures and designs, and that is also technology agnostic, stood out as an urgent requirement. Not an easy one, but we are being pragmatic about it: we’ll start with example diagrams on a basic custom component library.

We can immediately apply that visual language to the design of a data platform at the scale of SMEs because many of our prospects and customers fall in that category. Not only startups, but clearly also incumbent industrial players are understanding the rewards of data and/or the cloud.

To conclude, we consider architecture an integral part of our engineering expertise, so we have to engrain it in our internal guilds by actively participating and contributing to our internal body of knowledge.