Coeo

4 December 2017

The top 10 considerations when designing a data platform

Designing a data platform for a new product or service is a fascinating and exciting challenge. It’s also a daunting one. Instead of plugging a new feature into an existing platform that is proven and reliable, the onus is on you. You have to start from scratch and create a platform that becomes just as proven and reliable.

Like I said: daunting.

From our wide experience of helping clients in the retail, financial services and gaming sectors, here are the top ten factors we always consider.

1. New technology

This doesn’t mean implementing every possible new feature or design methodology. Instead, it’s about gaining the knowledge required about the latest technologies available so that the most appropriate decisions are taken when building the foundation of a solution. Existing functionality may already be offered as part of a third party system or service, for example, which may eliminate the need for development while providing a cost-effective, managed, secure and scalable option.

2. High cohesion and low coupling

A design in which functionally can be separated with relative ease into components performing specific tasks will enable a far greater flexibly. The ability to replace or adapt parts of an application will allow for quicker re-architecture and for third party solutions to be incorporated.

3. Testing

The ability to test in an automated and repeatable manner the full scope of the design is absolutely critical, as without it there is an increase in both unknowns and risk.

4. Resource

The limitations of physical resource will affect the design more than any other factor. Inconsistent network communication or high latency links, for example, will make architectures that perform high volume or synchronous transactions impossible to support.

5. Financial

As design complexity increases, there is often a point where it becomes subject to financial restrictions that make further development or expansion no longer viable.

6. Technical

It’s important to understand the technology being implemented and how it can be used. A design that pushes the limits of a technology can work in many cases, but thought must be given to how far a design is pushed and the restrictions that are present. For example, real time synchronization across a global infrastructure may quickly hit limitations.

7. Legislative

A design may achieve many of the core requirements of a business in redundancy and high availability by an infrastructure which is globally dispersed, but this may be impracticable because of legal requirements to have data hosted in geographical locations.

8. Scalability

Deciding how a design will scale is almost as important in many ways as the functional ability of the solution. A solution designed with little thought for scalability usually relies on vertical scale-up strategy that can very quickly hit both resource and financial limitations. A horizontal scale-out solution, however, which incorporates sharding or data partitioning will scale to far higher levels as the shared load is easier to scale.

9. Security

A system that may seem incredibly secure today will be obsolete tomorrow as technology and research is undertaken. It’s important to make sure that security is implemented at a level that meets the requirements, but it’s prudent to exceed those requirements up until the point where it becomes impossible in terms of resource. A secure system is always most vulnerable at the weakest point and implementing minimal security only guarantees that.

10. Planning

Plan how to design the platform using an iterative approach to ensure goals are achievable and well thought out. A plan should be based on what needs to be achieved and not what time allows. If a plan is shaped by additional factors, these should be highlighted and risks identified.

Coeo’s platform consultants can help you define and design a data platform that is operationally right for your business. Find out more about us.