Data Sync Document

For

VARStreet

By

VARStreet India Pvt. Ltd.

VARStreet Inc. VARStreet Software (I) Pvt.

Ltd.

2620, Augustine Drive, Suite Marisoft-1, 6th floor, Marigold,

174, Santa Clara, CA 95054 Wadgaonsheri, Pune- 411014,
India.

Phone: 1 (408) 562-5634 Phone: +91 - 20 - 6410 4118

Fax: 1 (408) 986-1196 Fax: +91 – 20 - 6607 2003

Website: www.varstreet.com
Document History

Document Name : Data Sync Document

Client Name : VARStreet Internal

Project Name : Data Sync Document

Version Number : 1.0

Prepared By : Mr. Yogesh Kandekar

Date : February 1, 2018

Approved By : Mr. Santosh Shilwane

Approval Date : February 1, 2018

Submitted By : Mr. Yogesh Kandekar

Submission Date : February 1, 2018

Revision Records

Sr. Version Release Reference Brief Description of Changes

No. Date Page
Numbers
1.0.0 Draft Version released for Internal Review
1. --
 Table of Contents

1. DATA SYNC (ORACLE TO MYSQL).......................................................................................................4

1.1. Location and Type..................................................................................................................4
1.2. Database Tables.....................................................................................................................4
1.3. Log Files and Location............................................................................................................4
1.4. Process Flow..........................................................................................................................5
1.4.1. About Data Sync.............................................................................................................5
1.4.2. Check log in oracle Database.........................................................................................5
1.4.3. Update Sync Request.....................................................................................................6
1.4.4. Data Sync Processing Dependency.................................................................................6
1.5. Steps to restart the Process...................................................................................................7
1.6. Technical Details....................................................................................................................8
1.6.1. Data Collect:...................................................................................................................8
1.6.2. Data Process:.................................................................................................................9
1.6.3. Scheduled Task on server:...........................................................................................11
1.6.4. Roadmap pending Items:.............................................................................................12
1.6.5. Frequent Issues or Enhancements:..............................................................................12
1. Data Sync (Oracle to MySQL)
1.1. Location and Type
Server IP Type Path on Server Source control Log File Path
Path

205.140.208.71 Batch E:\NewCatalogProcess\ $/ E:\

File DataSyncProcess_Catalog_Var VSSearchCompon NewCatalogProcess\
Part ent/ DataSyncProcess_Cata
VSSearchCompon log_VarPart\Log
ent.DataSync

205.140.208.71 Batch E:\NewCatalogProcess\ $/ E:\

File ContractDataSync VSSearchCompon NewCatalogProcess\
ent/ ContractDataSync \
VSSearchCompon Log
ent.
ContractDataSync

1.2. Database Tables

 DATASYNCREQUEST

1.3. Log Files and Location

For Data Sync:

Log files are generated for every entity of Data Sync for every day.

 E:\NewCatalogProcess\DataSyncProcess_Catalog_VarPart\Log

1) For Custom Products system will create file as: CustomProduct_Date.txt

Ex. CustomProduct_13-08-15.txt
2) For Store Catalog system will create file as: NewCatalog_Date.txt
Ex. NewCatalog_13-08-15.txt
For Manufacturer in store catalog system will create file as:
Manufacturer_Date.txt
For Categories in store catalog system will create file as: Categories_Date.txt
For Usage Group in store catalog system will create file as: UsageGroup_Date.txt
For Products in store catalog system will create file as: Products_Date.txt
3) For Pricelist system will create file as: PriceList_Date.txt
Ex. PriceList_13-08-15.txt
4) For Distributor system will create file as: Distributor_Date.txt
Ex. Distributor_12-08-15.txt
5) For Configuration system will create file as: Configurator_Date.txt
 Ex. Configurator_10-08-15.txt

For Contract Product Sync:

 E:\NewCatalogProcess\ContractDataSync\Log
System is creating log file as: log_Date.txt

Ex. log_5_8_2015.txt

1.4. Process Flow

1.4.1. About Data Sync

It is process which pick request from DATASYNCREQUEST table and process according to
REQUESTTYPE.

We have following request types:

a. Store Catalogs - CATALOG

b. Custom Products – VARPART
c. Price Lists - PRICELIST
d. Distributors - DISTRIBUTOR
e. Configurations – CONFIGURATOR
f. Contract Products - CONTRACTPRODUCT

1.4.2. Check log in oracle Database

If anyone wants to check sync log of any request then he can query to DATASYNCREQUEST
table as, if we want to retrieve today’s requests for varcompany 6036 then we can query
as:

SELECT * FROM DATASYNCREQUEST WHERE RESELLERID = 6036 AND TRUNC

(ENTEREDON) = TRUNC (SYSDATE) ORDER BY LASTUPDATEDON desc;

Also we can give additional parameters are per specifications.

For checking current process status we have STATUS column in DATASYNCREQUEST.

# Status Code Status Description

1 0 PENDING
2 -1 INPROCESS
3 1 SUCCESSFUL
4 2 FAILED
 For new request STATUS is 0, for request in process it is -1 for successful request it is
1 and for failed request it is 2.

If we need any additional from relevant table then REFERENCEID column will be
helpful. For Sore Catalog request it is CATALOGLISTID, custom product it have VARPARTID,
for pricelist it have PRICELISTID. So we can use this column for relevant data checking in
parent table.

Reference Id and reference Name relations:

# Reference Name Reference Id Table Name

1 CATALOG CATALOGLISTID VARCATALOGLIST
2 CONFIGURATOR CONFIGURATORID VSXCCONFIGURATOR
3 CONTRACT CONTRACTID CONTRACT
4 CONTRACTPRODUCT CONTRACTPRICEID CONTRACTPRICE
5 DISTIPRODUCT PARTID -
6 DISTRIBUTOR VARDISTRIBUTORID VARDISTRIBUTOR
7 PRICELIST PRICELISTID VARPRICELIST
8 STORE STOREID V_VSXCSPSTORE
9 VARPART VARPARTID VARPART
10 WEEKLYPROCESS RESELLERID -

1.4.3. Update Sync Request

If anyone wants to update failed request having status=2 them he can update status=0, so
that record will process again.

UPDATE DATASYNCREQUEST SET STATUS = 0, STATUSREMARKS = null, FAILEDCOUNT =

null WHERE RESELLERID = ????? AND TRUNC(ENTEREDON) = TRUNC(SYSDATE) AND
REQUESTID = ?????? ;

1.4.4. Data Sync Processing Dependency

If you need to process any request manually then you need to check dependency of
processing as:

# TYPE PRIORITY

1 VARPART 1

2 CATALOG 2

3 PRICELIST 3

4 CONTRACTPRODUCT 4

5 WEEKLYPROCESS 5
 If we have all above request for one company id then first we should process
VARPART as so on.

1.5. Steps to restart the Process

 Scenario 1 No. of Pending Request greater than 100

 Check for no of request in DataSyncRequest table. It should not be more than 100.
 If request for REQUESTTYPE=’VARPART’ is more than 100, and exe for DATASYNCVARPART is
running and critical request comes to process then follow the steps as:
 First Stop that exe, update status=-1 records to status=0.
 For all request for REQUESTTYPE=’VARPART’ update status=3 and for critical request update
status=0.
 Run schedule task for DATASYNCVARPART, it will process and once it done then check status
of critical request, it must be 1, if it is 2 then you need to check log file for that request.
 Now update status=3 requests to status=0.
 Run again schedule task for DATASYNCVARPART, to process remaining all request as well.

 Scenario 2 REQUEST < 100

 Check for no of request in DataSyncRequest table.

 Check any exe for VARPART, CATALOG, and PRICELIST is running, if any exe is already
running then wait till it over.
 Check request status and make it to 0.
 Once no exe is running then you can manually run exe that you need to process.

After done check status of request it must be 1, if it is 2 then check log file for that request.
1.6. Technical Details
1.6.1. Data Collect:
(1)In Data Sync we have data collect for PriceList, Catalog, Store
Profile, Distributor and Configurator.
(2)For Custom products we have trigger “triggervarpartinsert” which
insert record in DataSyncRequest for each updates.
(3)For Weekly Process we insert records on Saturday morning and run
process.
1.6.2. Data Process:

(1) We have data Process call for each process and it will run verify
records in datasyncrequest table and process accordingly.
(2)In this process we are taking data from oracle for that request
using referenceid and fill it using DTO.
(3)DTO will used to fill data for parameters in MySQL procedure and
call procedure to insert data in MySQL respective tables.
(4)For weekly process we internally call “refresh prices” exe.
b) Data Processes:

i) VARPART: It is used to sync custom products to mysql from oracle.

System will update existing and add new.
Process include:
(1)Sync of custom catalog
(2)Sync of custom catalog specifications.
(3)Sync of bundle and bundle components and specifications.

ii) PRICELIST: It is used to load pricelist from oracle it will create new if
not exists or update if exists and it also create custom and standard
rules in mysql.
Process include:
(1)Sync of pricelist
(2)Sync of standard price rule.
(3)Sync of custom price rule.

iii) CATALOG: It is used to sync store catalogs to mysql from oracle.

System will update existing and add new. And sync category,
manufacturer, product and usage group rules.
Process include Syncing of:
(1)catalog definition
(2)Sync of standard manufacturers.
(3)Sync of custom manufacturers.
(4)Sync of standard categories.
(5)Sync of custom categories.
(6)Sync of standard products.
(7)Sync of custom products.
(8)Sync of usage groups.

iv) VARDIST: It is used to load distributors from oracle it will create new
if not exists or update if exists.

v) CONFIGURATOR: It is used to sync custom products to mysql from

oracle. System will update existing and add new.
Process include:
(1)Sync of configuration definition
(2)Sync of default components.

vi) EXPIREDVARPART: It is check if any custom product exispring today

then it will add entry in datasync to delete that product from mysql.

vii) MSP: It is used run MSP creation logic for given company id.

viii) VARPARTSPEC: It is used to load custom product specifications to

mysql.
 ix) STORE: It is used to sync store profile to mysql from oracle. System
will update existing and add new.

x) WEEKLYPROCESS: It is used to run weekly process on provided

reseller or it will take resellers as per thread and executes weekly
process, it will check dependencies as stated above.

1.6.3. Tables Mapping:

# TABLE IN ORACLE TABLE IN MYSQL

1 VARPART varproduct_master, product_master, custom_product, category,

manufacturer, varcategory, varmanufacturer

2 VARCATALOGLIST tcdb_mdm.mdo_catalog, catalog_category, catalog_manufacturer,

catalog_product, catalog_usagegroup

3 VARPRICELIST tcdb_mdm.mdo_pricelist, pricelist

4 CONTRACTPRICE contract_product, contract_category, contract_manufacturer

1.6.4. VARPART Flow:

1.6.5. VARPART Process:
1. Process is used to sync VARPART data to MySQL tables of VARProduct_master,
Product_Master, custom_products, category and manufacturers.
1.6.6. Scheduled Task on server:

1.6.7. Roadmap pending Items:

 a) Daily process in data sync.
b) Contract sync merge in Data sync project
c) Duplicate methods and code optimization.

1.6.8. Frequent Issues or Enhancements:

1. VARPART mandatory columns blank.

2. Add new columns in VARPART Sync if present in oracle.

3. Catalog manufacturer include/exclude issue, need to verify data.