PASS Data Community Summit logo

PASS Data Community Summit 2023

Get all the latest announcements direct to your inbox

IT Compliance and Software Development

What is IT Compliance and is it really necessary for contemporary Agile applications to be constrained by the requirements of compliance? William Brewer argues that if the objective is rapid delivery of applications, then compliance controls must be understood as early as possible in development.

DevOps, Continuous Delivery & Database Lifecycle Management
Automated Deployment

Bad things happen to businesses. Because of this, organisations are required to be resilient and to be good custodians. When I use the word ‘required’, I mean that businesses have, in many aspects, no legal alternative: Enterprises simply aren’t allowed to be reckless, otherwise the public would suffer, and this is why there are numerous regulations to protect the public. Official organisations must comply with the relevant statutory regulations about resilience, however lucky they might feel. If they don’t, then they face prosecution. They also will probably want to adopt a company policy of complying with self-regulatory industry codes, the recommendations of trade associations and the doctrines of the organisation. They will certainly need to comply with contractual obligations.

Compliance with the statutory regulations doesn’t guarantee that your organisation will, in reality, be secure, resilient and responsible. IT in general, and IT Security, in particular, is such a fast-moving science that the regulatory process can’t keep up. It is merely a ‘backstop’, to make sure that the minimum is done, that you meet your legal obligations and that the procedures are in place. For IT people, Compliance is just part of the business requirements: if meeting business requirements is important to you in your professional role, then compliance is important.

The scale of the task

The bulk of the work of compliance will apply to regulatory standards such PCI DSS , ISO 27001, HIPAA, HIPAA, HITECH, FISMA, SOX, or ‘glibba’ (Gramm-Leach-Bliley) as well as regional, national and international privacy regulations. In addition to the minimum legal standards, (such as the data protection act in the UK), there are international standards and guidelines that are recommended but are not enforceable. There are many of these, and they are evolving. There are many regulations on business continuity: The ISO22301 standard describes general business continuity practices whereas ISO27031 refers to IT practices. Recommendations for Disaster recovery are set out in ISO/IEC 24762. Most countries have separate and more detailed requirements for public bodies, especially primary responders. In the UK, this is in the Civil Contingencies Act 2004. Hospitals are another special category in most western countries, and any industry where failure results in death, injury or financial loss, such as mass transportation will have specialised legislation to protect the public.

Frameworks to summarise the legislation

Governments such as the United States or Europe will create frameworks that interpret existing standards and pre-existing ‘safe harbour’ documents into a document that provides guidelines for the IT practices that are likely to ensure compliance for their own government agencies (e.g. NIST Special Publication 800-53 for security risk management for federal information systems).

Compliance within an organisation

The office-holders within an organisation, directors, trustees, board-members or whatever, are responsible for compliance, and have a legal obligation to ensure that it happens . This means that they can be held personally liable if their organisation fails to meet the requirements of statutory regulations. As well as obvious matters such as the accuracy of company financial reports, and the internal control processes used in the generation of these reports, they can also face severe penalties for lapses in Business Continuity, Information Security and Custodianship. The practical implementation of these obligations is usually, for larger organisations, delegated to an activity within the organisation that ensures that rules and policies are adhered to, and that this adherence can be proven. This is usually done by an auditing team appointed by the organization. If a company is of sufficient size, it will appoint a compliance specialist who has the task of making sure that all the appropriate people been informed of the policies, and that they have, where possible, been implemented and checked.

A company must be able to prove at any time to the public and shareholders that they comply to all required relevant regulations. To do this, they have to be able to show that policies have been developed, put in place, and followed to meet all identified risks, usually verified by professional auditors. Unless good centralised auditing systems are in place, this can be a very costly and time-consuming process, and the compliance function in any organization will therefore aim to make audit as easy as possible.

Establishing a framework of IT policies and controls

IT compliance  is just one aspect of the overall e task of compliance that  any organisation faces, but it is an important part.

Not only must businesses comply with a wide range of regulations and obligations, but they also have to prove that they comply. In general, an organisation will try to avoid duplication of effort and activity by adopting a framework of policies that help members of the organisation to make sure that their work complies. To do this, the fog of legislation and contracts is sifted through. From these are distilled the IT control requirements specific to the legislation that is in-scope for their organization and for the various categories of data that they manage. This consolidated and harmonized sets of compliance controls are then adopted as policy, sometimes within a compliance framework, and implemented. A system of audits is then devised in order to ensure that the organization complies at the correct level. This task of distilling all the current legislation and obligations into a framework is important since legislation is often confusing, paradoxical and contradictory. It is also almost always out-of-date. It is generally better to get this work done centrally in any organisation and then just comply with the distilled framework.

Of major interest to IT are policies that cover

  • business resilience in the face of disasters,
  • regulations that cover how long, and und under what circumstances, data on individuals can be held
  • good custodianship, meaning preventing sensitive data passing into the wrong hands.

As well as these major concerns of Business Continuity, Information Security and Custodianship, IT must be assiduous in record-keeping. It is important to be able to do ‘forensic work’ to determine what went wrong and to learn from them. It must keep a reliable record of activity of sufficient quality that it can be used in evidence for any civil or criminal legal action.

An organisation is obliged to prove that it is acting responsibly in holding data, and taking all appropriate steps to ensure that it is handled appropriately. The  confidentiality, availability and integrity (CIA) of corporate information assets and intellectual property is more important for the long-term success of organisations than traditional, physical and tangible assets, and so it is important for the ‘stakeholders’ of the organisation as well as the general public and other entities that trade with the organisation to be reassured that these major concerns are attended to .

Business Resilience Standards For IT Applications

The bad things that happen to IT applications within businesses are, by their very nature, unexpected. They can include natural disasters, economic disruption and market turbulence, sabotage, terrorist-related incidents and disruption, cyber-crime and cyber terrorism, civil emergencies, strikes, and similar action, pandemic threats, disruptive technological advances, technology failure or supply chain failure. Most often, they are caused by incompetence, which only comes to light when the application is subjected to stress.

To avoid bad things happening to applications, they must be resilient, well-defended and actively monitored. Defence must be in depth, and requires obvious techniques such as security, access control, intrusion-detection and staff-education. IT General Controls, and application controls must be in place and checked regularly. Resilience involves actively identifying risks, mitigating those risks, and verifying that the risks have been mitigated over time. They must then be able to prove that they are adequately prepared, and have been able to respond appropriately in the past. To show that it is resilient, any IT application that is important for an organisation must have a continuity plan which fleshes out, and rehearses, a response to all identified and likely operational disruptions, and a disaster recovery plan which enables the organisation to recover from real disasters.

It is difficult to be prepared for the unexpected, but it relies on being able quickly identify any threat: The most effective strategies involve detecting unusual patterns of activity; particularly of access, and investigating them. The important processes must be monitored and checked for anything unusual or suspicious such as fraud or intrusion. These checks will involve audit reports, event logs, video, and version history. It will be necessary to allow investigators to track all events associated with a particular suspicious activity, and support staff must be given ways of countering hostile intrusions into the system. All forensics must be available and verifiable by an auditor. For a successful prosecution of the perpetrator of a fraud committed against the organisation, all monitoring data must be of the standards that allow it to be used as evidence.

Compliance of users of applications.

Even intricate security regimes can be rendered entirely useless by members of the organisation. A high proportion of ‘leaks’ of data come from staff. This can be carelessness or deliberate.

Deliberate leaking of data is best prevented by careful access-control: major incidents in the past were exacerbated by junior members of staff having wildly inappropriate access to highly-confidential materials. Not only did they have access but there were no controls in place to detect high levels of access, which would have alerted management to a problem.

Carelessness can be as devastating. It is important to adopt an organizational policy for employees using IT systems that is easy to understand. A well-drafted information security policy needs to cover the creation, transmission, transport and retention of information; It needs to define, for example, when secure VPN access must be used and what data can be stored on mobile devices, It would also have to cover the remote, wireless, electronic and physical access to the corporate network, and explain the dangers of hot-desking, using cloud-services, BYOD, Public WiFi and so on. Homeworkers require a whole new level of security audit since a VPN link can give a hacker broad access.

Compliance and IT Staff

To many people who are working in IT, the policies that are in place may seem somewhat bewildering, arcane and tiresome. They will affect development work, software purchase, network design, IT architecture and a smorgasbord of other IT tasks. However, although there is give and take over the interpretation of policies, they represent an organisational decision, taken in the face of a legislative decision as part of a wider democratic process. Arguing with the people who are tasked with turning policies into practice won’t change much, except perhaps their morale. Compliance with existing organisational policies is at the bedrock of understanding the business requirements.

Much of the typical regulatory framework is common-sense and enshrines current good practice. As well as providing a network with tested and proven disaster recovery and business continuity systems that include off-site backup, ops staff will monitor their network in real-time, and ensure high levels of security for their confidential enterprise assets, as well as providing network compliance audit reports to auditors when demanded. DBAs will have the delegated responsibility to mitigate all risks to data and the continuance of database systems in the event of a disaster, whether natural or man-made. As well as having the task of complying with the IT policies of the organisation. They will take guidance from the compliance experts on what the legal requirements are, and should be able to refer to the organisation’s current IT policies and IT control requirements.

Compliance and Application Development

One aspect of IT Compliance affects development work, because it aims to ensure that applications meet legal requirements, corporate policies, and industry standards. To do this, is usual for specialists to advise development teams on ways of making compliance easier, by summarising the task, advising on architectural designs that minimise the task, and keeping non-specialists up to date with security risks and the post-mortems of data breaches. The right time for this discussion to take place is early in development work, because certain aspects of legislation, such as data governance/security are hard to retrofit into an application after the major architectural decisions have been taken.

Although software suppliers will try to suggest that, by installing a package, it somehow provides immunity from compliance problems, this is simply untrue. Compliance is more about the way that an organisation functions, not about the way that applications function.

With the classic development methodologies, the compliance objectives that applications had to meet were part of the definition of the business architecture. By the time that development started, these were documented and so developers would be able to create the application in the light of these requirements. The compliance of the application could be checked before release. The problems here happen because the compliance criteria can get forgotten in the rush to meet release deadlines. When software delivery doesn’t happen because the compliance components aren’t in place, the compliance activity itself is often, perversely, blamed for the delay.

Because of Agile’s emphasis on the values of risk mitigation and continuous customer involvement, it should make it easier to ensure that the organisations requirements for compliance are met, but it also means that there has to be a higher involvement of compliance experts with the development process: The compliance equivalent to DevOps.

In Agile methodologies, there must be an up-front understanding in the team of what the compliance objectives are so that they become an intrinsic part of the Agile delivery culture. This would allow teams to include them as part of the Product Backlog as the acceptance criteria of the User Story. Where the compliance framework is in sufficient detail to identify the control activities, then tests can be automated to ensure that they are done Compliance, in Agile terms, is built up iteratively.

Compliance and Data

You must know where the data that requires most protection is located, and make sure that its special requirements are documented and understood.. Identify the different types of data held and make sure that personal, transactional, audit, financial and other sensitive data is zoned so that special data-retention conditions can be restricted to the data for which it is appropriate. There is no sense, for example, in being obliged to encrypt an entire database when there is only one column that is sensitive. It could be held perhaps on a special secure server along with other sensitive information.

The identification and classification of data is at the root of good custodianship, and is one of the most important activities of IT, yet the most commonly neglected.

Summary

Compliance with the existing regulatory framework, industry standards, guidelines and ethical standards is at the heart of good business. IT compliance is merely an important part of meeting the requirements of the business as closely as possible.

For any development team, whatever methodology it uses, it is important to have a grasp of the application controls and special requirements for data that are necessary for compliance and the instrumentation of the application, alongside the monitoring software, that is required for security. The auditing function for any application can never be added as an afterthought, but needs to be an intrinsic part of any application.

Compliance is often seen as a drag on development, stifling creativity and change. I don’t agree: It is better to see it as a timely warning that business requirements are never simple and take time and effort to elucidate from a mass of sometimes conflicting information. The nettle must be grasped as early as possible in development because controls to meet objectives of security, audit and instrumentation, for example, are hard to fit retrospectively after the code has been cut.

DevOps, Continuous Delivery & Database Lifecycle Management
Go to the Simple Talk library to find more articles, or visit www.red-gate.com/solutions for more information on the benefits of extending DevOps practices to SQL Server databases.