It used to be a considerable task to creating a feed for a data warehouse. Now, it just takes a few minutes to work through a series of screen s to create a pipeline that, in this example, creates a pipeline that brings data from a remote FTP server, decompresses the data and imports the data in a structured format, ready for data analysis. The Copy Wizard for the Azure Data Factory is a great time-saver, as Feodor Georgiev explains.… Read more
If you need a way of deploying infrastructure-as-code to Azure, then Azure Resource Manager (ARM) Templates are the obvious way of doing it simply and repeatedly. They define the objects you want, their types, names and properties in a JSON file which can be understood by the ARM API. Ed Elliott takes the mystery out of a simple means of specifying your Azure environment, whether it is a VM with blockchain software, SQL Server or a Web App on Linux with PostgreSQL
… Read more
Developers enter their trade fizzing with untamed creative energy. They envision spending their days dashing out intricate algorithms that compile first time, developing creative applications that dazzle colleagues and users alike, before leaving the office each evening, with a swagger in their step, to work their new GitHub ‘side project’, which will one day soon … Read more
How does one get a truly random sample of data of a certain size from a SQL Server database table. Well, there are simple non-portable tricks one can use, such as the NewID() function, but then refining those can be tricky. Take the Rand() function for a start. Can it really provide you with a truly random number? Why doesn't the TABLESAMPLE clause give you a set number of rows? Joe Celko scratches his head a bit, explains some of the issues and invites some suggestions and tricks from readers.… Read more
Azure Data Lake Store is an extendable store of Cloud data in Azure. You can move data to and from Azure Data Lake Store via Azure data Factory or Azure SQL Database and connect to a variety of data sources. You can script upload files from on-premise or local servers to Azure Data Lake Store using the Azure Data Lake Store .NET SDK. Ambily KK shows how easy it is to get started by setting up a script or App to move data to Azure Data Lake Store.… Read more
It is when you use R in SQL Server with one of the huge range of packages that comes with it that you can begin to appreciate the power of the system. With a package such as ggplot there are many 'knobs one can twiddle' in order to get spectacular and informative visualisations. Rob Sheldon continues his beginners series for R in SQL Server by showing how to refine the output to get it as you need it.… Read more
Many large organisations are compelled to embark on wide-ranging business-transformation IT projects. They are difficult to manage because, in the intervening months before the end of the project, the changing business environment will have forced further changes on the business. Agile holds out the promise of managing these changes more easily. Mohammad Rizvi explains, from experience, some of the the issues you are likely to face, and how you can solve them.… Read more
Azure Data Factory provides a radical new cloud-based way of collecting and preparing data in preparation for its storage and analysis. How do you get started with it to explore the possibilities it provides? Feodor Georgiev shows the practicalities of how to go about the task of preparing a pipeline for use, from preparing the Azure environment to downloading a file from a FTP to a blob storage in the Azure environment… Read more
Python is new to SQL Server 2017. It is intended primarily to allow the use of Python-based machine-learning within SQL Server, but it can be used for far more than this, with any Python libraries or Frameworks. To provide an example of what is possible, Hitendra shows how to use the feature securely to provide intelligent application caching, where SQL Server can automatically indicate when data changes to trigger a cache refresh. … Read more
Whenever you require a visit to the doctor, or hospital, a lot of personal data will likely be recorded, alongside details of your condition, the treatment you required, drugs prescribed and so on. At some point, you might have signed a consent form, one of the clauses of which allowed your data to be analyzed … Read more
If your development team needs to work on anonymised copies of the current production database, and if changes are being delivered rapidly as well, that could mean a lot of time and routine DevOps work copying databases. SQL Clone was designed for tasks like this. Grant Fritchey investigates whether you save time, effort and scripting over the more traditional approach, and at what point it makes sense to use it.… Read more
It is surprising that so much can be identified by deduction from data. You may assume that you can safely distribute partially masked data for reporting, development or testing when the original data contains personal information. Without this sort of information, much medical or scientific research would be vastly more difficult. However, the more useful the data is, the easier it is to mount an inference attack on it to identify personal information. Phil Factor explains.… Read more
Entity Framework (EF) is designed to work with a variety of data sources. Although this presents many advantages, there is a downside that many of the special features of a data source such as SQL Server are off-limits. Query Hints are an example: though often misused, they are occasionally important. Dennes Torres shows how you can use these in EF, using a command interceptor that will allow you to use any query hint with SQL Server.… Read more
There are several obvious problems with poor SQL Coding habits. It can make code difficult to maintain, or can confuse your team colleagues. It can make refactoring a chore or make testing difficult. The most serious problem is poor performance. You can write SQL that looks beautiful but performs sluggishly, or interferes with other threads. A busy database developer adopts good habits so as to avoid staring at execution plans. Rob Sheldon gives some examples.… Read more
Microsoft (StreamInsight), and Azure Stream Analytics represent a very different model for processing data. They are concerned with processing complex event streams of data (CEPs) from such things as sensors to deduce significant patterns and apply filters. Joe Celko discusses the background to an intriguing technology of complex event processing to establish the difference between data at rest, and data on the move.… Read more
