{"id":101539,"date":"2024-02-09T15:17:01","date_gmt":"2024-02-09T15:17:01","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=101539"},"modified":"2024-03-25T16:11:00","modified_gmt":"2024-03-25T16:11:00","slug":"automating-bcp-with-powershell","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/sysadmin\/powershell\/automating-bcp-with-powershell\/","title":{"rendered":"Automating BCP with PowerShell"},"content":{"rendered":"<p>I was talking with a good friend that has an environment with terabytes of information and to create a homolog or dev environment to him is a pain. He comes to me with a solution : A PowerShell script that gets only the first 1000 lines of each table, creates the inserts and schema files and runs in another server. I said to him. It\u00b4s crazy. We already have a solution that does that, BCP. So why not automate BCP? He agreed with me and then this article is how we did that . When you finished to read the article you will be able to understand how apply this solution if its is your case or even for not too large databases it\u00b4s a very delightful way to perform this operation.<\/p>\n<h2>The Problem<\/h2>\n<p>We have tons of data that needs to be exported only the first 1000 lines in a text file. This process needs to be automated and with less human intervention. Doing all repetitive tasks by writing code is what separates programmers from the non-programmers.<\/p>\n<h2>The Solution<\/h2>\n<p>As mentioned, the solution is to automate BCP as much as possible, and as the title belies, I am using my automation tool of choice: PowerShell. Like a lot of tasks you want to automate, this solution is not 100% automated.<\/p>\n<p>Often you reach a point where the cost of writing the code is greater than some of the manual work. It is something you need to consider when automating processes. How many times you do a task * number of times the task is done needs to be compared to the costs to automate. In this case I did the same, so you will see some manual parts of the process, but I guarantee that if you need to do something similar, it will save a lot your time. Certainly, enough to take the time to do a little automation.<\/p>\n<p>It\u00b4s simple and consist of 3 PowerShell scripts :<\/p>\n<ul>\n<li><code>ExportTablessmo.ps1<\/code> \u2013 It\u00b4s the script that will export all the tables (<em>and only the tables<\/em>) from the source database to the target database. You may also need to disable foreign key constraints in the target database before the load because BCP is not smart enough to deal with that.\n<p>If you have schemas and user defined types (or any other objects that make dependencies) I suggest you use SSMS to script out these objects before apply the script tables (of course). The code I am using does not create these parts of the target database for you.<\/li>\n<li><code>Bcpoutsmo.ps1<\/code> \u2013 This script is responsible to export out the data from the tables in the specified database inside the script (source database)<\/li>\n<li><code>Bcpinsmo.ps1<\/code> \u2013Finally this one imports the exported tables into the target database<\/li>\n<\/ul>\n<p>We need to create on SQL Server a login to do the BCP work. I will use a SQL Standard login named <code>BCPExportLogin<\/code> with a password of <code>ExPoRtPassword<\/code><em>!<\/em> in both servers (source and target). For some reason I could not make BCP works with windows authentication. For your production work you should use a user with a much better password and the least rights needed, then drop the login as soon as you have completed the task.<\/p>\n<h2>The scripts<\/h2>\n<p>First let\u00b4s create the sysadmin login. Open the SQL Server Management Studio and execute the following (note the code for this and all of the files is available <a href=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/LaerteJunior_PowerShellBCP.zip\">here<\/a> in a file named <code>CreateLogin.sql<\/code>, and there is a <code>DropLogin.sql<\/code> as well):<\/p>\n<pre class=\"lang:tsql theme:ssms2012-simple-talk\">USE [master]\r\nGO\r\nCREATE LOGIN [BCPExportLogin] WITH PASSWORD=N'ExPoRtPassword', \r\n   DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF\r\nGO\r\nALTER SERVER ROLE [sysadmin] ADD MEMBER [BCPExportLogin] \r\nGO<\/pre>\n<p>Both machines will need to have SMO installed. Fortunately, it comes as part of the SSMS installation, so you typically won\u2019t need to add anything else. There are other methods, which you can find in <a href=\"https:\/\/learn.microsoft.com\/en-us\/sql\/relational-databases\/server-management-objects-smo\/installing-smo\">this page on the Microsoft Learn site<\/a>.<\/p>\n<h3>ExportTablesSMO.ps1 \u2013 Used to export the table structures.<\/h3>\n<p><em>Note that the comments for the code follow the actual code and correspond to the #&lt;Letter&gt; tags.<\/em><\/p>\n<pre class=\"\">[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | out-null #A\r\n\r\n$sourcedatabaseName=\"adventureworks2016\" #B\r\n$sourceserverName=\"DESKTOP-SQEVVO1\\SQL2017\" #C\r\n$baseExportPath=\"c:\\Export\" #D\r\n\r\nNew-Item -Force $baseExportPath -type directory #E\r\n\r\n$Server = new-object ('Microsoft.SqlServer.Management.Smo.Server') $Sourceservername #F\r\n  ($server.databases[$ourcedatabaseName] |\r\n     select -ExpandProperty tables).script() |\r\n       out-file \"$baseExportPath\\scripttables.sql\" #G<\/pre>\n<p>The letter comments are explained in the following:<\/p>\n<ul>\n<li>#A \u2013 loads the SMO assembly<\/li>\n<li>#B &#8211; creates the source database name variable<\/li>\n<li>#C \u2013 creates the source server name variable<\/li>\n<li>#D \u2013 creates the variable to the folder to export the tables. The file created will be scripttables.sql<\/li>\n<li>#E \u2013 creates the folder based on #D<\/li>\n<li>#F &#8211; connects to the source SQL Server<\/li>\n<li>#G \u2013 connects to the source database, get the tables and then script it and save into a file<\/li>\n<\/ul>\n<h3>BCPOutSMO.PS1 \u2013 Exports the data from the database into files<\/h3>\n<pre class=\"\">[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | out-null #A\r\n$SourcedatabaseName=\"adventureworks2016\" #B\r\n$SourceserverName='DESKTOP-SQEVVO1\\SQL2017'#C\r\n$baseExportPath=\"c:\\Export\\tables\" #D\r\n\r\nNew-Item -Force $baseExportPath -type directory #E\r\n\r\n$Server = new-object ('Microsoft.SqlServer.Management.Smo.Server') $SourceserverName #F\r\n$server.databases[$SourcedatabaseName] | #G\r\n    select -ExpandProperty tables | #H\r\n      ForEach-Object { #I\r\n             $expression = \"bcp \"\"select top (1000) * from $_\"\" queryout $baseExportPath\\$_.txt `\r\n                   -c -U BCPExportLogin -P ExPoRtPassword -S $SourceserverName -d $SourcedatabaseName\" #J\r\n\r\ninvoke-expression $expression #L\r\n}<\/pre>\n<ul>\n<li>#A- loads the SMO assembly<\/li>\n<li>#B &#8211; creates the source database name variable<\/li>\n<li>#C \u2013 creates the source server name variable<\/li>\n<li>#D \u2013 creates the variable to the folder to export the content of the tables. Each file created will be the name of the table<\/li>\n<li>#E \u2013 creates the folder based on #D<\/li>\n<li>#F \u2013 connects to the source server<\/li>\n<li>#G \u2013 selects the source database<\/li>\n<li>#H \u2013 Select the tables from this database<\/li>\n<li>#I \u2013 creates a foreach to the tables<\/li>\n<li>#J \u2013 Build the BCP string<\/li>\n<li>#L \u2013Invoke the BCP string<\/li>\n<\/ul>\n<h3>BCPInSMO.PS1 \u2013 Imports data into the database<\/h3>\n<pre class=\"\">[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SqlServer.SMO') | out-null #A\r\n\r\n$newdatabaseName=\"newdatabase\" #B\r\n$newserverName='DESKTOP-SQEVVO1\\SQL2017' #C\r\n$baseExportPath=\"c:\\Export\\tables\" #D\r\n\r\n$Server = new-object ('Microsoft.SqlServer.Management.Smo.Server') $newserverName #E\r\n\r\n$server.databases[$newdatabaseName] |#F\r\n        select -ExpandProperty tables | #G\r\n          ForEach-Object { #H\r\n               $expression = \"bcp $_ in $baseExportPath\\$_.txt -c -U BCPExportLogin -P ExPoRtPassword `\r\n                    -S $newServerName -d $newdatabaseName\" #I\r\ninvoke-expression $expression #J\r\n}<\/pre>\n<ul>\n<li>#A- loads the SMO assembly<\/li>\n<li>#B &#8211; creates the target database name variable<\/li>\n<li>#C \u2013 creates the target server name variable<\/li>\n<li>#D \u2013 creates the variable pointing to the folder will load the files<\/li>\n<li>#E \u2013 connects to the source server<\/li>\n<li>#F \u2013 selects the source database<\/li>\n<li>#G \u2013 Select the tables from this database<\/li>\n<li>#H \u2013 creates a foreach to the tables<\/li>\n<li>#I \u2013 Build the BCP string loading the tables in the foreach<\/li>\n<li>#J \u2013Invoke the BCP string to load the data<\/li>\n<\/ul>\n<h2>Putting it to work<\/h2>\n<p>In this section I will demonstrate the process of using these scripts.<\/p>\n<p><strong>Step 1 \u2013 Export the schema<\/strong><\/p>\n<p>To export the schema table of the source database use the script <code>ExportTableSMO.ps1<\/code><em>. <\/em>The variables to change to your environment are <code>$sourcedatabasename<\/code>, <code>$sourcesevername,$basexport<\/code>.<\/p>\n<p>The output will be a file called <code>scripttables.sql<\/code> in the <code>$baseexport<\/code> path ?<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"875\" height=\"61\" class=\"wp-image-101540\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101539-1.png\" \/><\/p>\n<p>If you open the <code>scripttables.sql<\/code> script that was generated, you will see something like the following, based on the database you have chosen:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"727\" height=\"536\" class=\"wp-image-101541\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101539-2.png\" \/><\/p>\n<p>Use this file to create the objects in your target database, typically using SSMS.<\/p>\n<p>As mentioned, you may need to manually create schemas, user defined types, and perhaps any scalar functions used in constraints.<\/p>\n<p><strong>Step 2 \u2013 Export the Data From Tables.<\/strong><\/p>\n<p>Open the <code>BCPOutSMO.ps1<\/code> script<em>. <\/em>Change the <code>$sourcedatabasename<\/code>, <code>$sourcesevername,$basexport<\/code>. To match your environment. Execute the script and you should see a set of .txt files created in the <code>$baseexport<\/code> folder. One for each table to be exported:<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"985\" height=\"401\" class=\"wp-image-101542\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101539-3.png\" \/><\/p>\n<p><strong>Step 3 \u2013 Load the data into the new database<\/strong><\/p>\n<p>Open the <code>BCPinSMO.ps1<\/code> script and change the header variables to meet your environment<em>. <\/em>The variables to change to your environment are <code>$newdatabasename,$newservername,$baseexportpath<\/code>.<\/p>\n<p>Set the <code>newservername<\/code> and <code>newdatabasename<\/code> to the target of your data. Point the <code>$baseexporpath<\/code> to the folder where the <code>.txt<\/code> files are from step 2. Then just run the script and the tables will be filled.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" width=\"1345\" height=\"611\" class=\"wp-image-101543\" src=\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/2024\/02\/word-image-101539-4.png\" \/><\/p>\n<h2>Conclusion<\/h2>\n<p>As we can see, it is fairly simple to set up this set of code that will save a huge amount of time if you have to do this process more than a few times. If this is done frequently, with a bit more work the entire process could be built into a solution that calls the code in each file with just a little bit of work.<\/p>\n<p>Any time you find yourself doing repetitive management tasks, consider if PowerShell can help you achieve what you need, if only the parts are especially cumbersome like importing and exporting data from a database.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I was talking with a good friend that has an environment with terabytes of information and to create a homolog or dev environment to him is a pain. He comes to me with a solution : A PowerShell script that gets only the first 1000 lines of each table, creates the inserts and schema files&#8230;&hellip;<\/p>\n","protected":false},"author":221715,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[53,35],"tags":[4635,4151],"coauthors":[6819],"class_list":["post-101539","post","type-post","status-publish","format-standard","hentry","category-featured","category-powershell","tag-powershell","tag-sql-server"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101539","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/221715"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=101539"}],"version-history":[{"count":3,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101539\/revisions"}],"predecessor-version":[{"id":101546,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/101539\/revisions\/101546"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=101539"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=101539"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=101539"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=101539"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}