You are on page 1of 47

RDP362

Advanced Installation and Configuration


for SAP Sybase IQ
Mark Mumy, SAP Big Data Architect
October 2013
Disclaimer

This presentation outlines our general product direction and should not be relied on in making a
purchase decision. This presentation is not subject to your license agreement or any other agreement
with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to
develop or release any functionality mentioned in this presentation. This presentation and SAP's
strategy and possible future developments are subject to change and may be changed by SAP at any
time for any reason without notice. This document is provided without a warranty of any kind, either
express or implied, including but not limited to, the implied warranties of merchantability, fitness for a
particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this
document, except if such damages were caused by SAP intentionally or grossly negligent.

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 2


Agenda

 IQ Overview
 Lab Prerequisites
 Lab 1 - Sybase IQ Demo Database
 Lab 2 - Connectivity
 Lab 3 – Creating Your New IQ Server and Database Instance
 Lab 4 – Creating a Schema within Your new Instance
 Lab 5 – Loading Data into your Table
 Lab 6 – Running Queries
 Lab 7 – Running IQ as a Service
 Q&A

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 3


SAP Sybase IQ Overview
What is IQ?
Sybase IQ Server

A Sybase IQ Server is a process running on a host machine


This database server is the ‘brain’ of Sybase IQ system
Users connect to a IQ server to access data in an IQ database
Can be custom configured

Designed to support only one Sybase IQ database


Multiple databases on a single server are not recommended
Customers may incorporate multiple databases into a single IQ database and maintain logical
separation of objects

Uses Sybase Adaptive Server Anywhere (ASA) features


Some ASA server code is used to support Sybase IQ
Details are on the next slide

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 5


SQL Anywhere and Sybase IQ

SQL Anywhere (SA) is a relational database system that can exist with or without Sybase IQ
ANSI SQL plus additional functionality
Proven database technology

Sybase IQ
Patented storage technology
Vertical data storage
High-performance query engine

Sybase IQ uses some of the features of ASA


Connectivity
Query parsing
Security
Component Integration Services (CIS)

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 6


Column Storage Processor

Conventional Database
• Data is stored horizontally
1 2 3 4 5 6 7 8 9 …
r1 • Querying without indexes and views is extremely
r2 I/O intensive
r3
r4
• Building indexes and views is a huge time and resource
r5 drain
• Database footprint must be dramatically expanded to
SAP SYBASE IQ make the environment efficient for querying
• Data is stored vertically — each column is stored
1 2 3 4 5 6 7 8 9 …
r1
separately
r2 • The data is the index
r3
r4
• Retrieve only columns used in the query —
r5 reduce system I/O dramatically
• Ad hoc queries handled painlessly
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 7
© SAP AG 2009. All rights reserved. / Page 7
Powerful column store indexing technology

• Most columns will have at least one


TYPE USAGE index
Fast Projection Compressed raw data • Index selection decisions based on
for result sets (Default) column cardinality (number of
Low Fast Low cardinality data (up to 1000 unique values)
unique values)
• Indexes and columns are stored
separately
High Non-Group Aggregation on the fly and range searches
• Multiple indexes used to resolve a
query
High Group Key fields and • Indexes are self maintaining
groupings for
cross-tabular
• No optimizer statistics to update
Date, Time, DT Date ranges, date part operations
• Indexes are compressed
Multi-Column Concatenated indexes Index Advisor :
Word Key word or phrase string searches Run sample queries and obtain index
building advice
Compare Column comparisons Index building is a relatively simple
exercise

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 8


SAP Sybase IQ Compression
Managing large amounts of detailed data
Conventional DBMS

Summaries
Aggregates
1 - 2 TB
Same INPUT Data: “Conventional DW” is
3x-6x larger than Sybase IQ DW
Indexes 2.4-6
0.5 - 3 TB TB

LOAD

1 TB
INPUT DATA: Base table
0.25 -
Aggr/Summ: 0 - 0.1 TB “RAW data”
Source: flat files, LOAD 0.9 TB Indexes: 0.05 - 0.3 TB no indexes
ETL, replication,ODS 0.9 - 1.1 TB
Base table: 0.2 - 0.5 TB

9 © 2013 SAP AG or an SAP affiliate company. All rights reserved. October 13 9


Sybase IQ Database

Includes the following components:


Three “Objects”
 Catalog store (for metadata)
 IQ store (for data)
 IQ temporary store (for temporary data)
 IQ system store (system area)
 IQ shared store (sharing data between nodes of a multiplex)
Server files
 IQ message log
 Catalog store transaction log
 IQ server logs

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 10


dbspace

A DBSpace is a logical collection of DBFiles

A dbfile is a physical/logical name for a database file


dbspaces may be a file system or raw partition allocated to Sybase IQ

A Sybase IQ database includes multiple dbspaces


The IQ store, catalog store and IQ temporary store all consist of dbspaces

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 11


IQ Store

Stores data as indexes compressed on disk


Contains the transaction log for data in the IQ tables
Structure to manage space allocation (free list)

There is one IQ store per database


Will usually consist of many dbspaces
These dbspaces can be built on raw partitions or file systems

The logical name is IQ_SYSTEM_MAIN for the first dbspace of the IQ store
First physical file name is dbname.iq (for file system devices)

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 12


Catalog Store

One catalog store dbspace per IQ database


Is always created on a file system
This dbspace grows dynamically as objects are added

Contains all information needed to manage an IQ database


Metadata stored in the system tables
Stored procedures (system and user)

Also known as the database file


The physical file names is: dbname.db

The logical name for the catalog store is SYSTEM

Continued …
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 13
IQ Temporary Store

Work area for the database


Sorting data to build certain indexes during loading
Query processing work area for sorting and grouping

Physical file name is dbname.iqtmp (for file system devices)

One IQ temporary store per database


Can consist of several physical files
Can be raw or file system
The logical name is IQ_SYSTEM_TEMP (for the first
IQ temporary store dbspace)

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 14


Storage Layout in IQ 16.0

IQ
Catalog Store
IQ Temp Store IQ Main Store 1 DBSpace
1+1 DBSpace Multiple DBSpaces Multiple dbfiles
Multiple dbfiles Multiple dbfiles

IQ_SYSTEM_MAIN The System Default DBSpace User Default DBSpace Another DBSpace
A DBSpace Defined by the option Defined by: Maybe READ-ONLY
<user>.DEFAULT_DBSPACE For archive storage
Can have Multiple dbfiles DEFAULT_DPSPACE May have 1 per user
Stores Free Lists, Header (Only 1 System Default DBSpace
Pages and Multiplex info But it can have multiple dbfiles)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 15
IQ Message Log

Readable log file for the IQ database that contains:


Insert and delete messages
Error messages
Status messages
Query plans

Physical file name is dbname.iqmsg

One IQ message log per database


File system only

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 16


Catalog Store Transaction Log

This is the transaction log for the catalog store


Not for the warehouse data in the IQ store

Used to roll back (undo) or roll forward changes as necessary

Physical file name is dbname.log

A small file which will slowly grow over time

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 17


IQ Indexes

Every column in an IQ table has at least one index used for data retrieval and query processing
A default index is created when you create a table
Additional indexes on a column may be needed for performance

Index type selection is be determined by:


Estimated cardinality of the data in the column
Column usage in queries

Ten different types of indexes


FP/n-bit (Default Index) DATE
LF (Low Fast) DTTM (Datetime)
HNG (High Non Group) TIME
HG (High Group) WD (Word)
CMP (Compare) TEXT

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 18


Putting It All Together: IQ Database/Server

Sybase IQ database components (single node)

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 19


SAP Sybase IQ Mulitplex

Full Mesh High Speed Interconnect


Virtual VDM1
Shared VDM2
CPU,
Shared
Logical Server 2 Memory
Logical Server 1
Virtual
SAN Fabric Shared
Storage
Virtual
Shared
Storage

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 20


SAP Sybase IQ Roadmap
Path to Actionable Intelligence
2009 2009 2010 2011 2011 1H 2013

XLDB
Analytics

MapReduce API

PlexQ MPP Foundation

Full text search; Web 2.0 API

In-database analytics

Very large database (VLDB) platform foundation

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 21


SAP Big Data
Real-Time and Real-Results
Key Components of World Class Big Data Portfolio

BIG DATA
BIG DATA BIG DATA
PLATFORM
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
+ APPS &
ANALYTICS + SERVICES
22
SAP Sybase IQ
A comprehensive analytics platform

Sybase PowerDesigner, Bradmark, SAS, SPSS, KXEN,


Sybase Replication Symantec, Fuzzy Logix, BMMSoft,
Server, Whitesands, Quest, Zementis, and SOLIX, and PBS
SAP and Panopticon and ZEND Visual Numerics

Optimized BI and EIM to Development and Packaged ILM


Ecosystem Predictive analytics
model and replicate administration tools applications
Application
services
Hadoop and
DBMS R

Comprehensive ANSI Built-in full-text In-database analytics with Big Data open
Web 2.0 APIs
SQL with OLAP search MapReduce and simulator source APIs

Most mature Comprehensive MPP queries, virtual marts, High-speed Structured and
column store lifecycle tiering and user scaling loads unstructured store

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 23


SAP Sybase IQ 16
What’s New!
SAP Sybase IQ 16 Engine
Comprehensive web-based Web-enabled analytics
monitoring and

Information lifecycle management


administration Communications and security Resilient Multiplex grid

Web based administration and


withstands network

Resilie
Role-based access control
interrupts and server failures

nt
LDAP authentication
Enterprise-grade RBAC
authorization and LDAP Query engine

Multiplex grid architecture


authentication Aggressive scale out

monitoring
Loading Hash partitioned tables Aggressively scaled out
High performance, fully engine and data affinity query engine
Fully
parallel bulk loading parallel Text search
In-database analytics
Intelligent query engine with
New indexing technology for data affinity for “shared
increased compression and fast, Column indexing N-bit and tiered nothing” performance on a
indexing
incremental batch loads subsystem flexible “shared everything”
Column store architecture
New Generation
Re-engineered column PETABYTE SCALE store
store for extreme data Low latency, write optimized store Low latency, concurrent write
volumes optimized delta store
Storage area network

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 24


SAP Sybase IQ 16
Innovations for extremely large databases (XLDB)

Storage Architecture Loading Engine


• New generation column store • Fully parallel bulk loading
• New partitioning and compression • Real-time loading into delta store

Petabytes SAP Sybase IQ Real-time


XLDB Analytics

System Reliability Query Processing


• Grid resiliency • Data affinity
• LDAP and role-based security • Aggressively parallel and distributed

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 25


SAP SYBASE IQ 16
NEW COLUMN STORE ARCHITECTURE
Before…
Value proposition Logical page
of fixed sized
Enhanced compression cells

Storage savings
Improved I/O bandwidth
offset = ((rowID – start rowID ) – 1 / cellsPerPage

Architectural considerations
Support variable number of cells per page NOW…
Support various page formats within a column Logical page cell values
of variable
High performance access paths sized cells

Even with variable length data, insert/update/delete efficiently


into an existing page
directory
Richer metadata

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 26


SAP SYBASE IQ 16
N-BIT DICTIONARY COMPRESSION

Value proposition
Reduced memory footprint Raw Data = 400 MB; 1 Billion 4-byte integer values fn (N)

Improved effective I/O rates N Token Size Savings

More efficient table scans 2 (1B * 2 ) / 8 = 250MB 93.75%


3 (1B * 3 ) / 8 = 375MB 90.6%
Reduced disk space 4 (1B * 4 ) / 8 = 500MB 87.5%
…….
24 (1B * 24 ) / 8 = 3000MB 25%
Architectural considerations
N-bit FP, instead of 1, 2 and 3-byte FPs
Different data pages for same column can have different values of “N” for N-bit 2->3
 No more requirement to rollover FP format for all column data 3->4
4->5
Column is N-bit by default, unless otherwise specified to be flat 5->6
6->8
Options provided to set threshold for rollover to flat (to prevent large dictionaries) 8->10
10->12
Options provided to prevent rollover to flat (to prevent long rollover time) 12->16
16->21
Compatibility mode allows the database to mimic IQ 15 rollover behavior 21->24

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 27


SAP SYBASE IQ 16
SMALL BATCH LOAD PERFORMANCE

Value proposition
Improved performance of frequent, small batch loads
Predictable performance of small batch loads:
 performance is proportional to the size of the data being
loaded, not the table being updated Periodic
Small tree
component Merge
Architectural considerations
Btree-
Trickle
Inserting into a large High Group (HG) b-tree index is insert
based
index
costly
HG index will have a tiered structure with a small tree
component and a large tree component
Small, batch loads into the HG index are written to the
small tree component quickly and synchronously
The small tree component is periodically merged into
the large tree component as a background task

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 28


SAP SYBASE IQ 16
FULLY PARALLEL BULK LOAD
Before…
Value proposition Serial and parallel two phase load process
Raw data and bitmapped indexes – partly parallel: mixture of horizontal
Improved performance and vertical processing
Maximize use of existing cores on the B-Tree based indexes (HG, TEXT. WD) – fully parallel
machine
Numerous bottlenecks:
Dynamic load balancing
• Complex thread scheduling
• Expensive synchronization points
Architectural considerations
• Steps that are executed too infrequently to
Load an index/column concurrently with keep threads busy
multiple threads
Remove all bottlenecks which contribute to NOW…
inefficient use of CPU and storage
Fully parallel two phase load process
Dynamically scale up and down degree of Raw data– fully parallel
parallelism depending on the workload All secondary indexes – fully parallel

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 29


SAP SYBASE IQ 16
HIGH VELOCITY DATA LOADING

Value proposition
Hybrid Table
Continuous analytics over operational data Storage
IQ DML Insert/Update/Delete
Write ahead/commit
High velocity, concurrent data modifications engine
Exploit large memory and core footprints In-memory RLV log on
RLV rows RLV Store shared store

Consistent View
Lock Manager
periodic merge
Architectural considerations global row
space
Write optimized in-memory In-memory RLV (Row-level IQ query
IQ main on
versioned) store engine
main rows shared store
Row level locking, and statement snapshot isolation
Low latency micro operations
Transaction Consistent View
In-memory RLV store has reduced compression, no Manager Manager
Version Manager

sorting, no indexing
Fully recoverable with dedicated transaction log
Asynchronous data transfer from In-memory RLV store to
IQ main store
Users choose which tables are In-memory RLV tables

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 30


SAP SYBASE IQ 16
QUERY SCALE OUT – HASH PARTITIONING

Value proposition
Gives best of both worlds of shared everything and shared
nothing
Decreases hardware needs and localizes processing

Architectural considerations
Data is automatically partitioned during loading with built-in
hash algorithms
Data is divided into persistent subsets
 Reduces results sharing
 More efficient CPU usage
 Reduces instantaneous temp usage
Optimizer will use hash partitions for join and group by when
available

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 31


SAP SYBASE IQ 16
LDAP AUTHENTICATION

Value proposition: Reduced TCO and improved security


Enable customers to hook into existing enterprise infrastructures
for managing users and passwords
Enable central management of password complexity policies
Multiple domains and multiple LDAP servers

Architectural considerations: WebServer


Secure communication with LDAP server using TLS
Deployable across various vendor’s Directory Service that
supports Lightweight Directory Access Protocol (LDAP)
Support 24x7 operation: Automatic failover and failback
Efficient design for frequent, short-lived connections
No client side changes needed
SQL Anywhere, Sybase IQ, and ASE can share common user
repository

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 32


SAP SYBASE IQ 16
MULTIPLEX ENHANCEMENTS

Shared Temporary Storage


 Reduces the size of local temporary store
 Simplifies sizing requirements for individual local stores
 Can use shared storage OR high speed network (10gb/sec+) to share data

Logical Server – Login redirection


 Zero changes on client side on dynamic changes to logical servers
 Single point to connection “redirector”
 HA with multiple servers in the connection string to prevent a single point of failure

Global Transaction Resiliency


 Suspend/resume global DML transactions, with a timeout, during INC disconnect/reconnect, coordinator
downtime

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 33


Prerequisites for IQ Labs
Sybase IQ Administration
Prerequisites (Windows)

On Windows Server/Workstation
 IQ 16.0 Installation
– Can be run directly from latest EBF
– Installation home directory should be <drive:>\sybase
– Can be performed by the ‘sybase’ login (must be a local administrator)
 Environmental Variables Instantiated
– Should be automatic when logging in by ‘sybase’ user
 Directory structure for Labs and Scripts
<drive:>\IQ_LABS
<drive:>\IQ_LABS\scripts
<drive:>\IQ_LABS\QPLANS
<drive:>\IQ_LABS\queries
<drive:>\IQ_LABS\scripts

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 35


Prerequisites (Windows)

Due to Windows UAC controls, IQ demo files, etc cannot be run from the Sybase home
directory
 C:\ProgramData\SybaseIQ\ contains user-changeable files

Information to Remember
 IQ Demo Instance Port Number: 2638
 Your New IQ Instance Port Number: 2648

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 36


Suggestions

Take time to examine scripts before execution

Multiple techniques are shown for running and administering IQ

Be aware of your path


 When running, paths are accessed from the point of view of the .db file in the server, even when running scripts
from a network client (unless using client file)

Remember to replace <machine> /<host> with the node name of your workstation/server when
necessary

Note: the ‘>’ (right caret) represents the command shell prompt and is not to be included in
commands

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 37


Suggestions

When cutting and pasting commands from this presentation:


 fix the ‘smart’ quotes to ensure matching quotes for commands (suggest pasting to notepad and re-copying)
 there may still be hidden characters that might force you to re-type the command anyway

It is suggested to always leave a command/terminal window open in the ‘C:\IQ_Labs’ directory


(Windows) or the ‘/home/sybase/IQ_LABS’ directory (Linux) for the duration of the labs to issue
commands from.

IQ Utilities on Windows and Linux (e.g. dbisql) can use the ‘\’ and ‘/’ interchangeably when
specifying paths.
 Suggest using ‘/’ in all paths in scripts and commands to prevent ambiguity and errors with escape sequences
– e.g. \n = newline

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 38


IQ Installation

Unpack the supplied IQ software to a working directory


 Ensure there are no spaces/blanks in your working directory path

(Windows) To install IQ, your login must be a local administrator


 You must still right-click on the ‘setup.exe’ and ‘Run as Administrator’
 Ensure you are installing to the C:\Sybase directory

On Linux, the ‘sybase’ user will perform the installation and run the binaries
 Run ./setup.bin
 Install to ‘/home/sybase’

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 39


IQ Installation

When installing IQ, perform a ‘Typical’ installation


 Install IQ to the directories described above for your platform
 Select to ‘Evaluate’ IQ (Select to ‘Install licensed copy’)
 Choose your location and agree to the license
 (Continue installation without license key)
 (Enterprise Edition/DT)
 Accept the default RMI port and service settings
 (use ‘sybase’ for SCC administrator password and SCC agent administrator password)
 Answer ‘No(Yes)’ to starting the SCC Server

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 40


IQ Installation

Post-Installation (Linux)
 Edit .bash_profile and add the following lines before the final ‘PATH=…’ command
>source /home/sybase/IQ.sh
>export LANG=en_US.UTF8
>export IQ_USE_DIRECTIO=1

 Save and execute ‘. .bash_profile’ to set the IQ environment variables (or logout/login)

Post-Installation (Windows)
 Open a command and type ‘set’ to review environment variables are set for ‘SYBASE’, etc.

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 41


IQ_LABS Installation (Windows)

Extract the IQ_LABS.zip to the C:\IQ_LABS directory


 IQ_LABS directory is in the .zip
 Extract the .zip to C:\

Copy the correct scripts for Windows


>copy C:\IQ_LABS\scripts\Windows\*.* C:\IQ_LABS\scripts\.
>copy C:\IQ_LABS\scripts\labsW.cfg C:\IQ_LABS\labs.cfg

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 42


Other Resources

Contact me via email at mark.mumy@sap.com

Check out my blog: http://scn.sap.com/people/markmumy/blog

Use the IQ Community on the SAP Community Site: http://scn.sap.com/community/sybase-iq

Use the IQ Users Group: iqug@iqug.org

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 43


SAP TechEd Virtual Hands-on Workshops and SAP TechEd Online
Continue your SAP TechEd education after the event!

SAP TechEd Virtual Hands-on Workshops SAP TechEd Online


 Access hands-on workshops post-event  Access replays of keynotes, Demo Jam, SAP TechEd
 Available January – March 2014 LIVE interviews, select lecture sessions, and more!
 Complementary with your SAP TechEd registration  View content only available online
http://saptechedhandson.sap.com/ http://sapteched.com/online

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 44


Feedback
Please complete your session evaluation for RDP362.

Thanks for attending this SAP TechEd session.


RDP362
Advanced Installation and Configuration
for SAP Sybase IQ
Mark Mumy, SAP Big Data Architect
October 2013
© 2013 SAP AG or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.
The information contained herein may be changed without prior notice.

Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.

National product specifications may vary.

These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and
SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth
in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.

SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and
other countries.

Please see http://www.sap.com/corporate-en/legal/copyright/index.epx#trademark for additional trademark information and notices.

© 2013 SAP AG or an SAP affiliate company. All rights reserved. 47

You might also like