You are on page 1of 22

UNSD - OECD Joint System

Data Transfer Mechanism - Technical Specifications UN Comtrade Version
Ref : 135664293.doc

UNSD - OECD Joint System Data Transfer Mechanism - Technical Specifications

UN Comtrade Version

Page ii

1 Feb 2006

INFO³OECD

Ref: 135664293.doc

doc .UNSD .OECD Joint System Data Transfer Mechanism .Technical Specifications UN Comtrade Version Page iii UPDATE FORM Name and Version number DST001C DISTRIBUTION Andreas Lindner Trevor Fletcher Grégory Legoff Lynda Hawe Ronald Jansen Markie Muryawan Henri Kayali Sent 1 Feb 06 OECD OECD OECD OECD UN UN Info³ By OECD/STD Signed Approved By Signed VERSION A B C DATE OPERATION SPECIFICS 19 Oct Creation of the document 05 12 Dec Add-ons: Comtrade Distribution details (operating mode) 05 1 Feb 06 Cover page and Introduction edited INFO³OECD Ref: 135664293.

.......................................................................................15 4.11 4......................................8 4..................................... DOWNLOAD DATA TO A CSV FILE............................................................................................1.....................................................5...........16 4........................UNSD ..7 3............... ABOUT THIS DOCUMENT..........................................................18 INFO³OECD Ref: 135664293.....................3........................................................6..................................................................................4.....................................................16 4............ ASSOCIATED DOCUMENTS............2 2....8 4................................................5... Check SQL Server Updates.........................6....................4...................................................................................................................................................................................................17 4.........................................................................................................6................................................................ Create the Destination Table.....................................................................................................................................2.................... USING THE CONFIGURATION FILE.......................... DATA TRANSFER PROTOCOLS......................................................................... Use ProjectID.................1.........................17 4..........3.........2............................................17 4......... OPERATING MODE....................14 4..................2 1..4.... DATA FLOW FORMATS........................................................................................6.................... WHAT IS MEANT BY “JOINT SYSTEM”?......5 3..............................................................6..................................................................... OVERVIEW.................................4 3........................1.........................................15 4...........................2...............................OECD Joint System Data Transfer Mechanism .............................................................. INSTALL COMTRADETOOLS...................................................................................9 4..........................12 4............................................................................................................................................4 3...........................................17 FIGURES...............1.......................................Technical Specifications UN Comtrade Version Page iv TABLE OF CONTENTS 1........2.. SECURITY......... INTRODUCTION.......................................4 3.........................................................................................................................................................................................................................................3 3...............................................................................6......................................................................................... TEST WEB SERVICES ......................... REQUIREMENTS................................5....6.............................6........................................................7...... Define the Column Mapping Between CSV File and Database Table......16 4............................................................... DATA STORAGE FORMATS...............16 4.................4 3................. IMPORT DATA INTO A SQL SERVER DATABASE............................... Data Capture............................................................... Change Column Mapping.................................................................... Import a Dataset into the Table – Debug Mode.......................................6...3....3..................................................................................... TECHNICAL SPECIFICATIONS..........................1....7 4...1 1......................................................................doc ....................................................................................5...2.................................................................................................................................. Define the Connection String............................................................................................................................................. Import Datasets....................................................5.......................................................................... Data Processing and Dissemination............................ Use the New ‘Resume’ Parameter........ DOWNLOAD COMTRADETOOLS SETUP FILES.....2...................2......4 3...............................6..............1..........................................2................................................1 1...................... SYNCHRONIZATION .................................6 3...

and (iii) computer applications for use in processing of trade data. memorandum items.1. (ii) developing additional analytical and presentational functionalities and controlling user access to them.doc .Technical Specifications UN Comtrade Version Page 1 1.g. Both organizations continue to keep separate (institutionally and legally) trade databases to ensure rights and ability of each organization to pursue their own policies with regard to (i) storing various data in addition to those agreed in Joint System [e. (ii) common trade data processing standards. This set of agreements. Introduction 1. The names of those databases are: • • For UNSD: UN Comtrade3 For OECD: OECD ITCS 4 Database To ensure user awareness of the joint efforts of both organizations in the area of international merchandise trade statistics. derived data etc].UNSD . both organizations agreed to make data responsibility very clear. and (iii) dissemination practices including pricing of user access to the database services. It is also to be used when referring to the respective databases. Any deviation from this practice is reflected in the country/period metadata. historical data. where UNSD and OECD jointly developed (i) an agreement on data collection and data sharing. entering into contracts with users.” Similarly. This considerable amount of very detailed investigation has led to the signing of an UNSD/OECD Memorandum of Understanding in January 2001 and further extensions. standards and applications is referenced by both organizations as: UN─OECD Joint Trade Data Collection and Processing System (Joint System) Both organizations agreed to use the above name in any documents describing their cooperative activities. trade data of OECD member countries are compiled by OECD in accordance with UN-OECD Common Trade Data Processing Standards and are available in UN Comtrade in accordance with UN/OECD Data Sharing Agreement. Consequently.OECD Joint System Data Transfer Mechanism . What Is Meant by “Joint System”? Since 2000. The Joint System is the foundation of the trade databases maintained by both organizations.. Any deviation from this practice is reflected in the country/period 1 2 United Nations Statistics Division Organization for Economic Co-operation and Development 3 Commodity Trade Statistics Database 4 International Trade by Commodity Statistics INFO³/OECD Ref: 135664293. etc. Each organization maintains its own brand name for use in data dissemination activities. the UN Comtrade disclaimer contains the following statement: “Beginning 2005. UNSD1 and OECD2 have joined forces to reduce the response burden for national data providers in the field of merchandise trade statistics and to agree on one unique and transparent set of data instead of having parallel (and often not harmonized) data collections. the OECD trade database disclaimer contains the following statement: “The trade data for the selected non-OECD member countries are compiled by UNSD in accordance with UN/OECD Common Trade Data Processing Standards and are available in OECD ITCS Database in accordance with UN-OECD Data Sharing Agreement.

doc . Associated Documents • • • • Data Capture / Data Processing Manual Web Browser Client User Manual Database Administrator Client Manual Database Dissemination Model Guide The UNSD – OECD Joint System package also includes the following documents: INFO³/OECD Ref: 135664293.2. Data transfers between UNSD and OECD are the responsibility of specialized OECD statistics database Administrators.Technical Specifications UN Comtrade Version Page 2 1.OECD Joint System Data Transfer Mechanism . Pictures include detailed comments in callouts or in their captions.UNSD . About this Document This document makes up the technical specifications of the Joint System data transfer mechanism (DTM) between UNSD and OECD databases. DTM is introduced in Section 2. below.3. at the beginning of the section. These comments contribute to trimming down the text and making it clearer. 1. This layout is generally appreciated by non-expert readers. Word links are available throughout this document –like for instance.

the technical specifications of data transfer as represented above: • • • • • • Transfer protocols Data flow formats Data storage formats Security OECD – UNSD synchronization Operating mode Some parts of this document are closely related to the Database Administrator Client Manual. step by step.Technical Specifications UN Comtrade Version Page 3 2.OECD Joint System Data Transfer Mechanism . Overview As mentioned in the previous section. In fact. The purpose of this document is to describe. This process is illustrated below: OECD Disseminate Working Working Database Database Data Exchange & Synchronisation UNSD Disseminate Dissemination Dissemination Database Database Dissemination Dissemination Database Database Working Working Database Database Process Common Common Processing Processing Module Module Process Common Common Processing Processing Module Module Data Access Data capture Data capture 5 ITS Meeting Sept 2005 Figure 1: statistical data flow for UNSD and OECD in Joint System Data transfers and conversions are physically carried out by an UNSD software package. one for each organization. the Joint System will still use two physical SQL Server databases. ComtradeTools functions. to store the collected and processed commodity trade statistical data. UNSD and OECD will use a unique logical storage to make their statistical data available through the Web. the databases will stay identical regarding their logical contents. particular additions and habits. named ComtradeTools . These two databases will be continuously updated and synchronized through a data exchange process explained later in this document. Thus. INFO³/OECD Ref: 135664293.doc . Configuration file and commands are discussed in the Operating Mode below. A specific “flavor” will also be added to each database contents to reflect OECD and UNSD standards.UNSD . which makes use of Web Services.

In fact. the OECD Client should have a basic knowledge of XML files and SQL Server Query Analyzer.NET Framework version 1. Importation into Access and SQL Server database are carried out through the Get External Data (Access) and Import Data (SQL Server) functions.2. 3. Excel or . checking.UNSD .1. Note: data in Access format is kept for archival purposes. Requirements The OECD Client who wishes to import updated data from the UNSD database should be equipped with the following software components: • • • • Microsoft Windows 2000 or a higher version. conversion.OECD Joint System Data Transfer Mechanism . WinZip program (note: some versions of Windows have a built-in unzip capability) In addition.Technical Specifications UN Comtrade Version Page 4 3. INFO³/OECD Ref: 135664293.1. SQL Server is used to store the statistical data to be processed (Working Databases) and distributed (Dissemination Databases) Data processing (importation. Microsoft .doc . Data Processing and Dissemination As mentioned before.2.2.1.DBF (FoxPro) files. Microsoft SQL Server 2000. any format is acceptable. Data Capture Contributing countries generally send their annual statistical data as plain text.2. MS Access is used as an intermediate storage for all these files before loading them up to the SQL Server database. Data Storage Formats 3. aggregation…) is described in the Data Capture / Processing Manual. 3. Technical Specifications 3.

SDMX.SDMX.3.g.org for more detail about SDMX.org/structure/key_families/cross/UN_COMTRADE_BE" xmlns:cross="http://www.OECD Joint System Data Transfer Mechanism .un. the contents of a plain text file.org/2001/XMLSchema-instance" xmlns:schemaLocation="http://www.SDMX. the XML5 format is used.xsd http://www.sdmx. During these transfers.SDMX.org/resources/SDMXML/schemas/v1_0/message" xmlns:uncs="http://unstats. The following insertion is an example of SDMX cross-sectional schema: <?xml version="1.001891" VALUATION="Special" TRADE_SYSTEM="CIF" PARTNER="Origin"> <uncs:Obs CC-BE="122" PRT="12" netweight="0" qty="0" QU="1" value="12990" /> </uncs:Section> </uncs:Group> </uncs:DataSet> </CrossSectionalData> - - - Figure 2: example of SDMX cross-sectional schema Link to http://www.org/resources/SDMXML/schemas/v1_0/cross SDMXCrossSectionalData.UNSD . The SDMX data is compressed by 70% to 80% before the data transfer.org/resources/SDMXML/schemas/v1_0/message SDMXMessage. The synchronization process is explained in this document and in the Database Administrator Client Manual.doc . The incoming data is finally converted into CSV7 format and loaded into the destination database. 1).xsd http://unstats.xsd"> <Header> <ID>UN626573874</ID> <Test>true</Test> <Truncated>false</Truncated> <Prepared>2005-11-10T04:34:38</Prepared> <Sender id="UN"> <Name xml:lang="en">United Nations</Name> </Sender> <KeyFamilyRef>UN_COMTRADE_BE</KeyFamilyRef> <DataSetID>01</DataSetID> <DataSetAction>Update</DataSetAction> <Extracted>2005-11-10T04:34:38</Extracted> </Header> <uncs:DataSet> <uncs:Group RPT="266" time="2004" CL="BE" UNIT_MULT="1" DECIMALS="1" CURRENCY="USD" FREQ="A" TIME_FORMAT="P1Y" REPORTED_CLASSIFICATION="H2" FLOWS_IN_DATASET="MX"> <uncs:Section TF="1" REPORTED_CURRENCY="XOF" CONVERSION_FACTOR="0.Technical Specifications UN Comtrade Version Page 5 3.org/resources/SDMXML/schemas/v1_0/cross" xmlns:xsi="http://www.un.org/structure/key_families/cross/UN_COMTRADE_BE UN_COMTRADE_BE_CrossSectional. 5 6 Extensible Markup Language Statistical Data and Metadata Exchange 7 Comma Separated Values.. Such a file can be used by Notepad or Excel INFO³/OECD Ref: 135664293. Data Flow Formats The OECD and UNSD databases are kept synchronized through bi-directional data transfers.w3. e. with SDMX6 cross-sectional schema.0" encoding="utf-8" ?> <CrossSectionalData xmlns="http://www. The same data format is used for transfers between working and dissemination databases (see fig.

due to data size. REST Web service is selected because of its ability to stream and use on-the-fly compression.UNSD . published on the Web via XML standards. In our case. The picture below illustrates the Joint System data transfer mechanism between UNSD and OECD: Figure 3: transfer mechanism between UNSD and OECD Transfers are “Pulled” by OECD: data is sent on OECD Users’ requests to ComtradeTools –see Operating Mode below. 8 Hypertext Transfer Protocol INFO³/OECD Ref: 135664293.Technical Specifications UN Comtrade Version Page 6 3. Data Transfer Protocols The common HTTP 8 is used all over the Internet via Web Services.OECD Joint System Data Transfer Mechanism .4. Web Services is a remote library (or Class. with properties and methods. from a language point of view).doc . some of which devoted to Web data transfers.

3. Indonesia (code 360) and its time stamp is transmitted by the keyword LastSynch (date of the last synchronization process). In the near future. In addition. its time stamp is updated to the current date. one week.OECD Joint System Data Transfer Mechanism .6. The IP address seen by ComtradeTools is the Web access provider’s. is described in the Data Capture / Data Processing Manual INFO³/OECD Co Co pr pr A A UNSD Timestamp (Comtrade Tools) OECD Ref: 135664293.g. The example below is the definition of such a key: <add key="2000Data" value="LastSynch=2004-01-06T18:08:46. and a comparison with the previous time stamp tells whether this data item must be downloaded to the OECD database to update it. The secured (encrypted) HTTP.synchronization process The synchronization process isn’t automated. Security It is based on IP Addresses of Users who wish to connect to the Web Server and on passwords to the dissemination database.UNSD . the UNSD data gathering and processing tool.. It selects a reporting country. The transfer/synchronization process is illustrated below: Reporting countries countries Reporting Figure 4: the import . the process will be automated and will take place. every night.doc .Synchoption=TIMESTAMP. Examples of such selection parameters are given in the next section. year or commodity. the User ID / password combination used when logging on to Windows is used to access the CoprA application (see Data Capture / Data Processing Manual). Its periodicity is left to the Administrator’s assessment – e. for instance. CoprA. Web Services is also used via the Access Control List function. HTTPS. More precisions about the Configuration file and its usage are given in the next section. could possibly be used. parameters=/r:360” /> The key is identified by its name. An example of denied access is given in section 4. not the local network address of the connecting station.Technical Specifications UN Comtrade Version Page 7 3.5. the updating data can be filtered by country. Synchronization The OECD and UNSD databases are synchronized through a time stamp mechanism inside ComtradeTools: each time a data set is imported from reporting countries to the UNSD database. in addition to being selected by the time stamp. ‘2000Data’. Selection criteria can be defined directly in the command line or by Keys in the ComtradeTools Configuration file. The format of a time stamp is YYYY-MM-DDTHH:MM:SS (labels in italics are placeholders) From a general point of view.

Depending on the status of the operating workstation and the data to be transferred. The easiest way to proceed is to choose Open. WinZip will be activated and will display the three expanded installation files (. a software package. System Drive is usually C). if not already present on the User’s workstation. all the useful operations are discussed. . In fact. is used to initiate the various transfer / conversion functions. The first thing to do is to download the ComtradeTools package.un. Operating Mode This section is an illustration of the operating procedure used to import UN data into the OECD database. some of the operations described here won’t be necessary to carry out an import.exe.1.ini. . As mentioned above. The conversion and synchronization phases are also reproduced. INFO³/OECD Ref: 135664293. the Registry Editor. for instance.org/unsd/comtradedist/comtradeToolsSetup. 4. ComtradeTools is a command line program: it is launched from the Run window.doc . like. An example of column mapping and SQL Server data creation is provided.zip The usual Open/Save interface is displayed.msi): Figure 5: the three ComtradeTools setup files – unpacked by WinZip Transfer the installation files to a proper local directory (suggested: <System Drive>:\Program Files\UNSD\ComtradeTools. Download ComtradeTools Setup Files Installation package (compressed file) Get ComtradeTools installation files by linking to http://unstats.OECD Joint System Data Transfer Mechanism . ComtradeTools.UNSD .Technical Specifications UN Comtrade Version Page 8 4.

Technical Specifications UN Comtrade Version Page 9 4.2. you just have to choose the Application folder (the installation folder defined during the previous step is the default).OECD Joint System Data Transfer Mechanism . Install ComtradeTools Double-click9 the . INFO³/OECD Ref: 135664293. in fact. The installation process takes a few seconds. The following files are created: Installation / Application folder ComtradeTools task Configuration file Figure 6: the ComtradeTools files (See next page) 9 Depending on the System options.doc . A wizard will help you install the Application.exe file.UNSD . a single click may be enough.

INFO³/OECD Ref: 135664293. as it is for any other application.OECD Joint System Data Transfer Mechanism .doc .UNSD .Technical Specifications UN Comtrade Version Page 10 The presence of the Application can also be checked through the usual Add/Remove Programs interface: Figure 7: Add/Remove Programs interface This utility should be used to uninstall ComtradeTools.

may need enlargement) The access control concerns the sender IP address as seen by the UNSD Server.aspx?px=H1&r=381&y=2003. the following message is sent: Figure 9: access to ComtradeTools function denied INFO³/OECD Ref: 135664293. The meanings of these keywords are explained below. If access is denied. If the transfer operation works fine.4. The selection parameters are identified by keywords like ‘px’ or ‘r’.org/unsd/comtradedist/getSdmxV1. For instance: http://unstats. Test Web Services To make sure that everything is in order –especially regarding access rights–.UNSD . as mentioned in the Security section.OECD Joint System Data Transfer Mechanism .2002&cc=TOTAL&p=0&comp=false This request asks for the transfer of a selected data set from the UN database to the OECD Client workstation.3. a request to ComtradeTools can be sent through the Web Browser.doc . the destination is a XML file with contents like: Figure 8: contents of the destination XML file after the transfer (partial view.un.Technical Specifications UN Comtrade Version Page 11 4. in section 4.

doc . Download Data to a CSV File The next requests will be sent from the Command Line ( aka DOS) window. ComtradeTools. This window pops up when selecting Start > Run.UNSD . a CSV file will be created with the converted imported data. Here.4. 9.Technical Specifications UN Comtrade Version Page 12 4. is followed by a list of slash-separated parameters. These parameters are: r: reporting country code. The following command is an example of download request: ComtradeTools /r:360 /y:2003 /px:H1 /action:DownloadAndConvertToCSV The program name. The resulting process can be followed on a DOS pane that pops up: Figure 11: command line execution – error(s) occurred We see that the transfer failed. Submitting the URL that appears in this DOS pane would return the access denied message in fig. 360 stands for Indonesia y: reporting year px: classification used. The Command Line window with the command (leftmost part) in it is reproduced below: Figure 10: command line window Clicking OK submits the transfer request to ComtradeTools. H1 means HS1996 action: here. (See next page) INFO³/OECD Ref: 135664293.OECD Joint System Data Transfer Mechanism .

for instance. with. The next records are the extracted data values. prefixed by the destination folder.txt’. ‘8’ as reporting country. INFO³/OECD Ref: 135664293. The name of this file is ‘Temph120018_CSV. The destination file name is built from the selection criteria: classification. We notice an extra parameter in the command line: outputDirectory with C:\Temp as value. The picture below shows the first records of the destination CSV file: Column headers Figure 13: dataset downloaded into a CSV file – first records The first line is made of column headers. the transfer would have been successful.OECD Joint System Data Transfer Mechanism . with the following messages on the DOS pane: Another command line Figure 12: command line execution – successful transfer The code of the reporting country is now 8 (Albania): Indonesia would have returned too large a data set. with the selected dataset values converted to CSV in it.Technical Specifications UN Comtrade Version Page 13 Had the access been granted. year and country.UNSD .doc . This is the local directory the CSV file is created in.

time=year.quantity=TradeQuantity" /> <add key="SQLConstValueMapping" value="source=UN" /> <!-. DownloadAndImportToSQLDB --> <add key="AttributeMapping" value="RPT=reporter.un.SynchOption=TIMESTAMP. updated automatically when SynchOption:TimeBased is selected--> <!-.value=TradeValue" /> <!-.org/unsd/comtradedist/refs/getDaTariffLine.aspx" /> <add key="XmlComtrade" value="http://unstats.Mapping from SDMX to CSV Header.org/unsd/comtradedist/getSdmxTariffLineV1.Database=EmptyComtradeDev.SQL Mapping from CSV Headers to Column Names plus the Constant Value.5.Parameters=" /> <add key="2003IndonesiaData" value="LastSynch=.classification=pfCode.aspx" /> <add key="DaTariffLine" value="http://unstats. CL=classification.doc .un.Parameters=/r:484" /> <add key="TFData" value="LastSynch=. qty=quantity.0" encoding="utf-8"?> <configuration> <appSettings> <!-.un.Technical Specifications UN Comtrade Version Page 14 4.org/unsd/comtradedist/refs/getDataAvailability.un. we’ll give examples of how the Configuration file can be used.SynchOption=TIMESTAMP.SynchOption=TIMESTAMP.org/unsd/comtradedist/getTariffLine.DateTime format is ProjectID=YYYY-MM-DDTHH:MM:SS --> <add key="2000Data" value="LastSynch=2004-01-06T18:08:46.reporter=rtCode.UNSD . Using the Configuration File This file has the XML format and contains the definition of Keys that are used by ComtradeTools as parameters.SQL Mapping from CSV Headers to Column Names plus the Constant Value.Time based Synch. CC-XX=commodity.un.Password=password.Web Service URL.TradeValu e=TradeValue. netweight=netweight. used only for action DownloadAndImportToSQLDB --> <add key="SQLColumnMapping" value="year=yr.Table=tempLoad" /> <!-.Server=(local). used only for action DownloadAndConvertToCSV.aspx" /> <add key="XmlTariffLine" value="http://unstats.QuantityUnit=qtCode.Parameters=/y:2003 /cc:TOTAL" /> </appSettings> </configuration> Figure 14: the original ComtradeTools Configuration file – notice the comments (HTML format) The keys in this file are defined by their names and values within an ‘add key’ tag. QU=QuantityUnit. INFO³/OECD Ref: 135664293.partner=ptCode.aspx" /> <!-.un. PRT=partner.Parameters=/r:360 /y:2003 /cc:TOTAL" /> <add key="2003Data" value="LastSynch=2004-04-27T15:37:01. It is part of the ComtradeTools package (see fig.netweight=NetWeight.aspx" /> <add key="DaComtrade" value="http://unstats.flow=rgCode.commodity=cmdCode.SynchOption=TIMESTAMP. used only for action DownloadAndImportToSQLDB --> <add key="SQLDestConnectionString" value="UserName=sa.org/unsd/comtradedist/get.aspx" /> <add key="SdmxComtrade" value="http://unstats.SynchOption=TIMESTAMP. The ‘2000Data’ key has been introduced in the Synchronization section. TF=flow. DO NOT CHANGE --> <add key="SdmxTariffLine" value="http://unstats.OECD Joint System Data Transfer Mechanism .org/unsd/comtradedist/getSdmxV1.Parameters=/y:2000" /> <add key="MexicoData" value="LastSynch=2004-11-06T18:08:46. 6) Its original contents (for the example we use) are displayed below: <?xml version="1. In the rest of this section.

and later when the CSV data is imported into the OECD SQL Server database. For instance.CONVERSION_FACTOR=cf. Change Column Mapping UN Comtrade Version Page 15 When the original UNSD data (SDMX format) is transferred to a CSV file.value=TradeValue" /> to: <add key="AttributeMapping" value="REPORTED_CURRENCY=cur.SynchOption=TIMESTAMP. LastSynch is reset and TIMESTAMP is used as synchronization option. netweight=netweight. The correspondence is defined by the two following keys in the Configuration file: AttributeMapping: SQLColumnMapping: SDMX to CSV CSV to SQL Server table In the Configuration file. we restore the two removed correspondences (reporter and year) in the AttributeMapping key. netweight=netweight. TF=flow.OECD Joint System Data Transfer Mechanism .UNSD . Finally. we’ll first change the AttributeMapping entry: <add key="AttributeMapping" value="RPT=reporter. INFO³/OECD Ref: 135664293.1. We can then run the transfer request again: ComtradeTools /r:8 /y:2001 /px:H1 /action:DownloadAndConvertToCSV OutputDirectory:C:\temp And compare the new CSV file we get with the previous one (fig. 2. CCXX=commodity. CCXX=commodity. PRT=partner. Use ProjectID Keys can be added to the Configuration file to replace a parameter string in the ComtradeTools command. 4.5. TF=flow. according to the parameters in IndonesianH1 and in the command line.doc . When the process is completed. 13). PRT=partner. we can insert the following definition in the Configuration file: <add key="IndonesiaH1"value="LastSynch=.2. SQLColumnMapping will be used later. time=year. QU=QuantityUnit.5. These keys (and therefore the parameter string) can be submitted via the ProjectID label in the command line.Technical Specifications 4. the timestamp will be updated to keep the synchronization process in order. The command line can now be: ComtradeTools /y:2003 /action:DownloadAndConvertToCSV /outputDirectory:c:\temp\sdmx\ /ProjectID:IndonesiaH1 The transfer and conversion to CSV will be carried out.value=TradeValue" /> We notice that: 1. time=year.Parameters=/r:360/px:H1" /> The name of the key is ‘IndonesiaH1’ and its contents make a selection by reporting country (360=Indonesia) and classification (H1=HS1996). qty=quantity. QU=QuantityUnit.CL=classification. a column mapping must take place between the source and the destination. two other correspondences have been added: REPORTED_CURRENCY=cur. qty=quantity. CL=classification. CONVERSION_FACTOR=cf. two correspondences have been removed: RPT=reporter.

4.[tempLoad] ( [pfCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . type=Character and length=50 or 255. [qtCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL .CustomTask failed. Reporting Country or Classification.Database=Northwind. Import Data into a SQL Server Database 4. the following error will occur: "Unhandled Exception: System.6. type and length. [source] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . N'IsUserTable') = 1) drop table [dbo].Server=(local).[tempLoad]') and OBJECTPROPERTY(id. if the table exists.InvalidCastException: QueryInterface for interface DTS. [TradeValue] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . [ptCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . [TradeQuantity] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL .[tempLoad] GO CREATE TABLE [dbo]. [cmdID] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . Northwind. Optional value parameters make it possible to customize the destination table according to Year. [yr] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . [cmdCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . it is removed from the database. the standard Windows demonstration database.6. [NetWeight] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . to create the destination table in a database (for this example.microsoft.exe Client component is installed on their workstation by linking to: http://www. Check SQL Server Updates The User should first make sure that the sql2ksp3. In this example.6.2. Create the Destination Table The following commands and source data should be submitted to SQL Server Query Analyzer. Define the Connection String The following key should be added to the Configuration file: <add key="SQLDestConnectionString" value="UserName=sdmx.Table=tempLoad"/> This is the path to the SQL Server database and the target table. INFO³/OECD Ref: 135664293. is used. Here.6.Password=sdmx.asp file If this component is missing. ‘tempLoad’ is then (re-)created and its columns are defined with their names.sysobjects where id = object_id(N'[dbo]. any existing database managed by SQL Server will do): if exists (select * from dbo.com/sql/downloads/2000/sp3.UNSD .1. either from the keyboard or from a script file. [rtCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL . this operation should be carried out only once for a given system. [rgCode] [varchar] (255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL .Technical Specifications UN Comtrade Version Page 16 4.3.OECD Joint System Data Transfer Mechanism .doc . [currency] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ." Of course. [conversion_factor] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ) ON [PRIMARY] GO Figure 15: SQL Query Analyzer input to create the destination table in a database This SQL sequence first checks for table ‘tempLoad’ existence. 4.

OECD Joint System Data Transfer Mechanism .2) We can now re-run the direct import with the synchronization option: ComtradeTools /action:DownloadAndImportToSQLDB /outputDirectory:c:\temp\sdmx\ /projectID:IndonesiaH1 /synchOption:Timestamp 4.cf=conversion_factor"/> Compared to the original value in the Configuration file. as defined in the Configuration File: ComtradeTools /r:8 /y:2001 /px:H1 /action:DownloadAndImportToSQLDB /removeTemp:false /DTSDebug:true With the DownloadAndImportToSQLDB action.5.6.year=yr. this is useful for troubleshooting. a new parameter in the command line makes it possible to resume the selected ComtradeTools action up to n times if errors are encountered.6. 4. UN Comtrade Version Page 17 Define the Column Mapping Between CSV File and Database Table Key SQLColumnMapping in the Configuration file is used. This action is carried out through SQL Server DTS (Data Transformation Services).classification=pfCode.7.flow=rgCode. Import Datasets The IndonesianH1 key has already been appended to the Configuration file: <add key="IndonesiaH1"value="LastSynch=.6. INFO³/OECD Ref: 135664293. Once the action is completed the User should check the contents of table tempLoad in the database.UNSD .Parameters=/r:360/px:H1" /> This key includes the reset of LastSynch (see section 4.4. In the original Configuration File.g. Refer to SQL Server documentation for details on DTS.quantity=TradeQuantity. we have: <add key="SQLConstValueMapping" value="source=UN" /> 4.5. The DTSDebug option makes it possible to create and examine the DTS package. the intermediate step that results in the creation of the CSV file is transparent to the User –its name is still built from the selection parameters. Import a Dataset into the Table – Debug Mode The ComtradeTools command line is used again to transfer data from the UN database and import the resulting CSV file into ‘tempLoad’ table in Northwind database.SynchOption=TIMESTAMP.partner=ptCode.6. the new columns ‘currency’ and ‘conversion_factor’ have been added. Its value should be: <add key="SQLColumnMapping" value="reporter=rtCode.netweight=NetWeight.cur=currency.6.doc .Tra deValue=TradeValue.Technical Specifications 4. Use the New ‘Resume’ Parameter As from 15 Aug 2005.. This parameter is used in the command below: ComtradeTools /action:DownloadAndImportToSQLDB /outputDirectory:c:\temp\sdmx\ /projectID:IndonesiaH1 /synchOption:Timestamp /resume:3 The operation will be resumed up to three times until no error occurs. labels) can also be mapped through the SQLConstValueMapping key. Constant values (e.commodity=cmdCode.

................7 FIGURE 5: THE THREE COMTRADETOOLS SETUP FILES – UNPACKED BY WINZIP..................................16 INFO³/OECD Ref: 135664293..............................................................................5 FIGURE 3: TRANSFER MECHANISM BETWEEN UNSD AND OECD...............8 FIGURE 6: THE COMTRADETOOLS FILES..........................................................UNSD ...........................11 FIGURE 10: COMMAND LINE WINDOW.......Technical Specifications UN Comtrade Version Page 18 FIGURES FIGURE 1: STATISTICAL DATA FLOW FOR UNSD AND OECD IN JOINT SYSTEM.......................................................doc .......................................................................................................................................................................................OECD Joint System Data Transfer Mechanism ..........................SYNCHRONIZATION PROCESS.10 FIGURE 8: CONTENTS OF THE DESTINATION XML FILE AFTER THE TRANSFER (PARTIAL VIEW.........................................................................................................13 FIGURE 14: THE ORIGINAL COMTRADETOOLS CONFIGURATION FILE – NOTICE THE COMMENTS (HTML FORMAT)...........6 FIGURE 4: THE IMPORT ......................................................................14 FIGURE 15: SQL QUERY ANALYZER INPUT TO CREATE THE DESTINATION TABLE IN A DATABASE..........................................................................................................11 FIGURE 9: ACCESS TO COMTRADETOOLS FUNCTION DENIED.....................................12 FIGURE 12: COMMAND LINE EXECUTION – SUCCESSFUL TRANSFER........................................................3 FIGURE 2: EXAMPLE OF SDMX CROSS-SECTIONAL SCHEMA............................13 FIGURE 13: DATASET DOWNLOADED INTO A CSV FILE – FIRST RECORDS........................................................................................................................................................................................ MAY NEED ENLARGEMENT).12 FIGURE 11: COMMAND LINE EXECUTION – ERROR(S) OCCURRED.................................................................9 FIGURE 7: ADD/REMOVE PROGRAMS INTERFACE........................................................................................