Exporting Data Programatically with bcp and xp_cmdshell - SQLTeam.

com

Page 1 of 3

Home|Weblogs|Forums|AskSQLTeam | SQLServer Links

Search:

Go

Active Forum Topics | Popular Articles | All Articles by Tag | SQL Server Books | About

Offering custom-written SQL Server classes focusing on real-world expertise. Classes never cancelled.

Exporting Data Programatically with bcp and xp_cmdshell

By Garth Wells on 01 July 2001 | 23 Comments | Tags: bcp/Bulk Insert

SubscribetoSQLTeam.com
Weekly SQL Server newsletter with articles, forum posts, and blog posts via email:

This article demonstrates how to programmatically control bcp to generate text files. It uses dynamic SQL and xp_cmdshell to execute a call to bcp from within Query Analyzer. In a previous article I showed you how to use BULK INSERT to import data stored in a text file into a table in SQL Server. BULK INSERT makes an "under-the-hood" call to bcp (Bulk Copy Program) to import the data. bcp is a command-line utility that can be used to import/export data in either character or native SQL Server format. Most developers use DTS (Data Transformation Services) to import/export data, but there are times when bcp is a more convenient approach. In this article I want to show you how to programmatically use bcp and xp_cmdshell to create a text file based on data stored in SQL Server. The approach shown here is used on a project I am currently working on to create text files during a nightly batch process. The batch process uses a stored procedure to accumulate daily transactions for EDI processing. More specifically, it retrieves and formats the daily transactions and then exports them to a text file which is ftp'd to the target vendors. The procedure creates several text files, which is why the approach shown in this article is more efficient than the one discussed in OSQL: Storing result of a Stored Procedure in a file

Subscribe
SQLTeam.com Articles via RSS SQLTeam.com Weblog via RSS

- Advertisement -

bcp in Action
Let's start off with a simple bcp call so you can see how it works. If you want to see the full syntax for bcp take a look the topic: bcp Utility, in Books Online (BOL). The following command exports the data stored in the authors table in the pubs database to authors.txt.

bcp "SELECT * FROM pubs..authors" queryout authors.txt -U garth -P pw -c

- Sponsor's Message -

To execute the statement, go to the Command Prompt and type it in as shown replacing "garth" and "pw" with a valid login/password. All we are doing is passing arguments to the bcp executable. The first argument is the SELECT statement that specifies the source data. The database name is referenced in the SELECT because by default bcp will try to run the statement in the default database associated with the "garth" login. If the default database associated with the login does not contain the target object(s), an error message is generated. The second argument (queryout) tells bcp the direction of data flow. In this case it indicates the data is flowing out of SQL Server using a query to specify the source data. The third argument (authors.txt) specifies the file that will store the data. The file will be created in the directory in which the command is executed if no other is specified. The fourth and fifth arguments (-U and -P) specify the login and password. The arguments are casesensitive, so make sure they are capitalized. You should know that the -T argument can be used in place of -U and -P when using a trusted connection. The last argument (-c) specifies the format of the data coming out of SQL Server. When -c is used the data is stored in character format using the tab character as the column separator and the newline character as the row terminator. After you execute the statement open authors.txt in Notepad and take a look at the format.

Become a Microsoft Certified Master (MCM) How do senior IT professionals differentiate themselves from the competition? The MCM program, through intensive training from world-class experts, plus Or call 866-462-8388 extensive written and lab(US and Canada) based testing, certifies only the most qualified candidates. As an MCM you'll join an elite community of Microsoft software and technology experts, and you'll be recognized as one of the top Microsoft technology experts in the world.

SQLShare.comVideos
Introduction to Deploying SSIS Packages
In this session, you'll learn the basics on how to deploy an SSIS package to the package store. You'll learn the benefits of whether to use the MSDB database or file system to store your packages and how to use configuration files to simplify migrations.

Using Dynamic SQL to Create a Valid bcp Call

The nightly batch process I referenced earlier creates date-specific files, so it is important to have the date the file is created in its name. Dynamic SQL is used to accomplish this. A file's name is created using the technique shown here.

Deployment Options for SQL Compact projects

This video takes a look at the different deployment options for a Visual Studio 2008 project using the SQL Compact edition database

DECLARE @FileName varchar(50) SET @FileName = REPLACE('c:\authors_'+CONVERT(char(8),GETDATE(),1)+'.txt','/','-') PRINT @FileName --Results-c:\authors_06-30-01.txt

Ranking Functions - Part 1

In this video Marco introduces all 4 ranking functions available in SQL Server. This is the 1st part, wich he shows how to use ROW_NUMBER. It might look simple, but ROW_NUMBER was a significant addition to TSQL, enabling us to cleanly assign numbers to rows when needed.

http://www.sqlteam.com/article/exporting-data-programatically-with-bcp-and-xp_cmdsh... 8.6.2010

Exporting Data Programatically with bcp and xp_cmdshell - SQLTeam.com

Page 2 of 3

The GETDATE() function is used to determines today's date, CONVERT changes the date into a more friendly format (e.g., 06/30/01) and REPLACE replaces the "/" characters with a "-". Once the file name is created, it is concatenated with the bcp command to create a valid call. The following shows how this is done.

Splitting Delimited Strings

It's not something we'll do often, but it's a useful tool to have - especially if you want to let Reporting Services users select multiple values and pass them to a stored procedure. We'll look at two techniques, one that uses the TABLE variable, and one that works all the way back to SQL 2000 by building an XML string and using OpenXML.

DECLARE @FileName varchar(50), @bcpCommand varchar(2000) SET @FileName = REPLACE('c:\authors_'+CONVERT(char(8),GETDATE(),1)+'.txt','/','-') SET @bcpCommand = 'bcp "SELECT * FROM pubs..authors ORDER BY au_lname" queryout "' SET @bcpCommand = @bcpCommand + @FileName + '" -U garth -P pw -c' PRINT @bcpCommand -- Results -bcp "SELECT * FROM pubs..authors ORDER BY au_lname" queryout "c:\authors_06-30-01.txt" -U garth -P pw -c

Performance Tuning Quiz #3 - Part 2

We're back with Part 2 of Quiz#3, showing you how to make SELECT COUNT(*) go faster against a simple table. As is often the case it turns out to be a case of needing a (better) index - did you figure it out? Did you get down to under 30 reads?

Resources
SQL Server Resources

Note: Results wrapped for readability.

Advertise on SQLTeam.com SQL Server Books SQLTeam.com Newsletter Write for SQLTeam.com Contact Us About the Site

Executing the Call with xp_cmdshell

Now that a valid bcp call has been constructed, we must execute it to create the data file. You have already seen how to do this interactively via the Command Prompt, but what we need is way to execute it within a stored procedure. This is accomplished with the extended stored procedure xp_cmdshell. Extended stored procedures are programs written in langauges like C++ that are then attached to an instance of SQL Server. Once attached, they can be referenced just like a regular stored procedure. xp_cmdshell allows you to execute an operating system command via a SQL Statement. Execute the following in Query Analyzer to see how xp_cmdshell works.

EXEC master..xp_cmdshell 'dir' -- Results (partial) -output -----------------------------Volume in drive C has no label. Volume Serial Number is 2876-AD19 NULL Directory of C:\WINNT\system32 NULL 06/27/01 08:38p <DIR> 06/27/01 08:38p <DIR> ...

. ..

The operating system command "dir" is executed and the results are shown in the Results pane of Query Analyzer. Note that I referenced the master database in the EXEC statement. To execute an extended stored procedure that starts with "xp_", either the master database must be active or referenced by EXEC. There are a few extended stored procedures that start with the "sp_" prefix. Those procedures, like all system stored procedures, can be executed without specifying the database because of the default behavior of SQL Server. When you execute a stored procedure that starts with "sp_", SQL Server looks for the procedure in the master database first. If the procedure is not found in master, it looks in the active database. If you have a stored procedure that you want to access from all your databases, create it in master and use a name that includes the "sp_" prefix. To create the data file we simply execute the command stored in @bcpCommand with xp_cmdshell. The following shows all the statements needed to create the file.

DECLARE @FileName varchar(50), @bcpCommand varchar(2000) SET @FileName = REPLACE('c:\authors_'+CONVERT(char(8),GETDATE(),1)+'.txt','/','-') SET @bcpCommand = 'bcp "SELECT * FROM pubs..authors ORDER BY au_lname" queryout "' SET @bcpCommand = @bcpCommand + @FileName + '" -U garth -P pw -c' EXEC master..xp_cmdshell @bcpCommand -- Results -output --------------------NULL Starting copy...

http://www.sqlteam.com/article/exporting-data-programatically-with-bcp-and-xp_cmdsh... 8.6.2010

Exporting Data Programatically with bcp and xp_cmdshell - SQLTeam.com

Page 3 of 3

NULL 23 rows copied. Network packet size (bytes): 4096 Clock Time (ms.): total 61 Avg NULL

2 (377.05 rows per sec.)

In the nightly batch procedure this code is referenced multiple times to create the various text files. For each reference, the SELECT is modified per the target tables and a new file name is generated.

Discuss this article: 23 Comments so far. Print this Article. This page has been read 245,460 times.
If you like this article you can sign up for our newsletter. We send it out each week that we post a new article. There's an opt-out link at the bottom of each newsletter so it's easy to unsubscribe at any time. Email Address:

Subscribe

Email This Subscribe to this feed Kick it Save to del.icio.us View blog reactions

RelatedArticles
Using BULK INSERT to Load a Text File (19 March 2001)

OtherRecentForumPosts
Need to hard code 2 customers (4 Replies) Special Character Issue (2 Replies) Export to Excel (495 Replies) need help for a query that contains variables (2 Replies) Trigger DELETED table columns-&gt; temptablevariable (2 Replies) complicated Select (10 Replies) Login failed for user ... Please, what am I doing (4 Replies) Sum/Ratio of other Sums (6 Replies)

2000-2010 SQLTeam Publishing, LLC | Privacy Policy

http://www.sqlteam.com/article/exporting-data-programatically-with-bcp-and-xp_cmdsh... 8.6.2010