Professional Documents
Culture Documents
This presentation outlines our general product direction and should not be relied on in making a
purchase decision. This presentation is not subject to your license agreement or any other agreement
with SAP. SAP has no obligation to pursue any course of business outlined in this presentation or to
develop or release any functionality mentioned in this presentation. This presentation and SAP's
strategy and possible future developments are subject to change and may be changed by SAP at any
time for any reason without notice. This document is provided without a warranty of any kind, either
express or implied, including but not limited to, the implied warranties of merchantability, fitness for a
particular purpose, or non-infringement. SAP assumes no responsibility for errors or omissions in this
document, except if such damages were caused by SAP intentionally or grossly negligent.
IQ Overview
Lab Prerequisites
Lab 1 - Sybase IQ Demo Database
Lab 2 - Connectivity
Lab 3 – Creating Your New IQ Server and Database Instance
Lab 4 – Creating a Schema within Your new Instance
Lab 5 – Loading Data into your Table
Lab 6 – Running Queries
Lab 7 – Running IQ as a Service
Q&A
SQL Anywhere (SA) is a relational database system that can exist with or without Sybase IQ
ANSI SQL plus additional functionality
Proven database technology
Sybase IQ
Patented storage technology
Vertical data storage
High-performance query engine
Conventional Database
• Data is stored horizontally
1 2 3 4 5 6 7 8 9 …
r1 • Querying without indexes and views is extremely
r2 I/O intensive
r3
r4
• Building indexes and views is a huge time and resource
r5 drain
• Database footprint must be dramatically expanded to
SAP SYBASE IQ make the environment efficient for querying
• Data is stored vertically — each column is stored
1 2 3 4 5 6 7 8 9 …
r1
separately
r2 • The data is the index
r3
r4
• Retrieve only columns used in the query —
r5 reduce system I/O dramatically
• Ad hoc queries handled painlessly
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 7
© SAP AG 2009. All rights reserved. / Page 7
Powerful column store indexing technology
Summaries
Aggregates
1 - 2 TB
Same INPUT Data: “Conventional DW” is
3x-6x larger than Sybase IQ DW
Indexes 2.4-6
0.5 - 3 TB TB
LOAD
1 TB
INPUT DATA: Base table
0.25 -
Aggr/Summ: 0 - 0.1 TB “RAW data”
Source: flat files, LOAD 0.9 TB Indexes: 0.05 - 0.3 TB no indexes
ETL, replication,ODS 0.9 - 1.1 TB
Base table: 0.2 - 0.5 TB
The logical name is IQ_SYSTEM_MAIN for the first dbspace of the IQ store
First physical file name is dbname.iq (for file system devices)
Continued …
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 13
IQ Temporary Store
IQ
Catalog Store
IQ Temp Store IQ Main Store 1 DBSpace
1+1 DBSpace Multiple DBSpaces Multiple dbfiles
Multiple dbfiles Multiple dbfiles
IQ_SYSTEM_MAIN The System Default DBSpace User Default DBSpace Another DBSpace
A DBSpace Defined by the option Defined by: Maybe READ-ONLY
<user>.DEFAULT_DBSPACE For archive storage
Can have Multiple dbfiles DEFAULT_DPSPACE May have 1 per user
Stores Free Lists, Header (Only 1 System Default DBSpace
Pages and Multiplex info But it can have multiple dbfiles)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 15
IQ Message Log
Every column in an IQ table has at least one index used for data retrieval and query processing
A default index is created when you create a table
Additional indexes on a column may be needed for performance
XLDB
Analytics
MapReduce API
In-database analytics
BIG DATA
BIG DATA BIG DATA
PLATFORM
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
+ APPS &
ANALYTICS + SERVICES
22
SAP Sybase IQ
A comprehensive analytics platform
Comprehensive ANSI Built-in full-text In-database analytics with Big Data open
Web 2.0 APIs
SQL with OLAP search MapReduce and simulator source APIs
Most mature Comprehensive MPP queries, virtual marts, High-speed Structured and
column store lifecycle tiering and user scaling loads unstructured store
Resilie
Role-based access control
interrupts and server failures
nt
LDAP authentication
Enterprise-grade RBAC
authorization and LDAP Query engine
monitoring
Loading Hash partitioned tables Aggressively scaled out
High performance, fully engine and data affinity query engine
Fully
parallel bulk loading parallel Text search
In-database analytics
Intelligent query engine with
New indexing technology for data affinity for “shared
increased compression and fast, Column indexing N-bit and tiered nothing” performance on a
indexing
incremental batch loads subsystem flexible “shared everything”
Column store architecture
New Generation
Re-engineered column PETABYTE SCALE store
store for extreme data Low latency, write optimized store Low latency, concurrent write
volumes optimized delta store
Storage area network
Storage savings
Improved I/O bandwidth
offset = ((rowID – start rowID ) – 1 / cellsPerPage
Architectural considerations
Support variable number of cells per page NOW…
Support various page formats within a column Logical page cell values
of variable
High performance access paths sized cells
Value proposition
Reduced memory footprint Raw Data = 400 MB; 1 Billion 4-byte integer values fn (N)
Value proposition
Improved performance of frequent, small batch loads
Predictable performance of small batch loads:
performance is proportional to the size of the data being
loaded, not the table being updated Periodic
Small tree
component Merge
Architectural considerations
Btree-
Trickle
Inserting into a large High Group (HG) b-tree index is insert
based
index
costly
HG index will have a tiered structure with a small tree
component and a large tree component
Small, batch loads into the HG index are written to the
small tree component quickly and synchronously
The small tree component is periodically merged into
the large tree component as a background task
Value proposition
Hybrid Table
Continuous analytics over operational data Storage
IQ DML Insert/Update/Delete
Write ahead/commit
High velocity, concurrent data modifications engine
Exploit large memory and core footprints In-memory RLV log on
RLV rows RLV Store shared store
Consistent View
Lock Manager
periodic merge
Architectural considerations global row
space
Write optimized in-memory In-memory RLV (Row-level IQ query
IQ main on
versioned) store engine
main rows shared store
Row level locking, and statement snapshot isolation
Low latency micro operations
Transaction Consistent View
In-memory RLV store has reduced compression, no Manager Manager
Version Manager
sorting, no indexing
Fully recoverable with dedicated transaction log
Asynchronous data transfer from In-memory RLV store to
IQ main store
Users choose which tables are In-memory RLV tables
Value proposition
Gives best of both worlds of shared everything and shared
nothing
Decreases hardware needs and localizes processing
Architectural considerations
Data is automatically partitioned during loading with built-in
hash algorithms
Data is divided into persistent subsets
Reduces results sharing
More efficient CPU usage
Reduces instantaneous temp usage
Optimizer will use hash partitions for join and group by when
available
On Windows Server/Workstation
IQ 16.0 Installation
– Can be run directly from latest EBF
– Installation home directory should be <drive:>\sybase
– Can be performed by the ‘sybase’ login (must be a local administrator)
Environmental Variables Instantiated
– Should be automatic when logging in by ‘sybase’ user
Directory structure for Labs and Scripts
<drive:>\IQ_LABS
<drive:>\IQ_LABS\scripts
<drive:>\IQ_LABS\QPLANS
<drive:>\IQ_LABS\queries
<drive:>\IQ_LABS\scripts
Due to Windows UAC controls, IQ demo files, etc cannot be run from the Sybase home
directory
C:\ProgramData\SybaseIQ\ contains user-changeable files
Information to Remember
IQ Demo Instance Port Number: 2638
Your New IQ Instance Port Number: 2648
Remember to replace <machine> /<host> with the node name of your workstation/server when
necessary
Note: the ‘>’ (right caret) represents the command shell prompt and is not to be included in
commands
IQ Utilities on Windows and Linux (e.g. dbisql) can use the ‘\’ and ‘/’ interchangeably when
specifying paths.
Suggest using ‘/’ in all paths in scripts and commands to prevent ambiguity and errors with escape sequences
– e.g. \n = newline
On Linux, the ‘sybase’ user will perform the installation and run the binaries
Run ./setup.bin
Install to ‘/home/sybase’
Post-Installation (Linux)
Edit .bash_profile and add the following lines before the final ‘PATH=…’ command
>source /home/sybase/IQ.sh
>export LANG=en_US.UTF8
>export IQ_USE_DIRECTIO=1
Save and execute ‘. .bash_profile’ to set the IQ environment variables (or logout/login)
Post-Installation (Windows)
Open a command and type ‘set’ to review environment variables are set for ‘SYBASE’, etc.
No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG.
The information contained herein may be changed without prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
These materials are provided by SAP AG and its affiliated companies ("SAP Group") for informational purposes only, without representation or warranty of any kind, and
SAP Group shall not be liable for errors or omissions with respect to the materials. The only warranties for SAP Group products and services are those that are set forth
in the express warranty statements accompanying such products and services, if any. Nothing herein should be construed as constituting an additional warranty.
SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and
other countries.