Published: November, 2012 Version: 5.0 Last Updated: December, 2013 www.qlikview.com
2
1 Whats New in Direct Discovery 11.2 SR5?
Direct discovery in SR5 has a number of changes and additions which have been categorized below:
1.1 Syntax
Old Syntax
New Syntax
DIRECT SELECT DIRECT QUERY EXPLICIT DIMENSION IMPLICIT MEASURE SQL() NATIVE() None available DETAIL None available DETACH
1.2 New and changed Set Statements Old Syntax
New Syntax
STAL after !5 seconds SET DirectCacheSeconds= 15; None available SET DirectStringQuoteChar=; None available SET DirectIdentifierQuoteChar=[]; SET LinkedConnectionMax= 4; SET DirectConnectionMax= 4; None available SET DirectTableBoxListThreshold= 100000; None available SET DirectDistinctSupport=false; None available SET DirectIdentifierQuoteStyle=ANSI;
1.3 Additiona !eat"res " #t is now $ossible to add a %&R clause to the scri$t for reload " 'lobal search on dimension fields now su$$orted " Added feature for drill to details with table bo(es and e(tra synta( )eyword * Detail " +ancel ,uery enhancement
3
2 #A$ 2.1 What is $i%&iew Direct Discovery? -li).iew Direct Discovery ca$ability combines the associative ca$abilities of the -li).iew in memory dataset with a ,uery model where the source data is not directly loaded into the -li).iew data model/ instead the aggregated ,uery result is $assed bac) to -li).iew user interface0 The direct discovery data set is still $art of the associative e($erience where the user can navigate both on the in*memory data and the direct discovery data associatively0
2.2 'ow is Direct Discovery di!!erent !rom the traditiona ("ery or data vis"ai)ation toos ca*a+iities? -li).iew Direct Discovery feature is a hybrid ca$ability where the in*memory and direct discovery data sets can be analyzed together/ even on the same chart0 The business users can associatively ma)e selections on either of the data sets/ and see what is associated and not associated with the same -li).iew association colours1 green/ gray/ and white0 They can create charts that would hel$ them analyze data from both data sets together0 This hybrid a$$roach $rovides much greater $ower and fle(ibility than the data visualization tools or traditional ,uery ca$abilities because with these tools the users can either create e(tracts to an in*memory engine or run ,ueries on the database but cannot do both on the same a$$lication $ersistently0 -li).iew Direct Discovery enables users to $erform business discovery and visual analysis against any amount of data/ regardless of size0 %ith the introduction of this uni,ue hybrid a$$roach/ users can associate data stored within big data sources directly alongside additional data sources stored within the -li).iew in memory model0 -li).iew can seamlessly connect to multi$le data sources together within the same interface/ e0g0 Teradata to SA2 to 'oogle 3ig -uery allowing the business user to associate data across the data silos0
2.3 'ow does $i%&iew manage the choice o! going to the data+ase vs. r"nning in memory? -li).iew decides which data resides in*memory and which data is direct discovery data by using s$ecial scri$t synta(0 This allows certain data elements dictated by the scri$t synta( not to be loaded into the -li).iew data model during the scri$t reload $rocess/ but still available for ,uery $ur$oses in -li).iew ob4ects in the user interface and to be combined for analysis with the -li).iew in memory dataset0 5nce this structure is established/ the direct discovery data can be 4oined with the in*memory data with the common field names0 This allows the user associatively navigate both on the direct discovery and in memory data sets0 -li).iew6s caching is also used to im$rove overall user e($erience0 -li).iew Server stores selection states of ,ueries in memory0 %hen the user ma)es the same set of selections/ -li).iew leverages the cache to im$rove the user e($erience0 #t is $ossible to set a time limit on caching0 2lease refer to ,uestion 780!9 for more information0
4
2., What are the +ene!its o! "sing $i%&iew Direct Discovery !eat"re? %ith Direct Discovery/ the business users can leverage the uni,ue -li).iew 3usiness Discovery ca$abilities on larger data sets0
Dri down to detais: The business users can drill down to details from the aggregated in memory data set0 They can leverage any data useful for analysis without scalability limitations of loading data in memory0 Associative data discovery on arger data sets: 3usiness users can analyze the larger data sets that they are not familiar with by ma)ing selections on the well*)nown data values that they use every day0 -li).iew will associatively show them the big data values for the selected in*memory data0 -xtend $i%&iews "ser ex*erience +eyond in.memory: Self*service 3ig Data analysis by levering the uni,ue -li).iew e($erience with social/ collaborative/ and mobile ca$abilities0 3usiness users can as) and answer ,uestions on their own and in grou$s and teams to forge new $aths to insight in the Data0 -asy to im*ement/ ra*id a** deveo*ment $i%&iew ex*erience: 3usiness users can achieve 3usiness Discovery on data sets that were $reviously used se$arately/ or not used/ because of their bul) and the develo$ment effort re,uired0 They can now analyze these data sets without waiting for the com$licated and lengthy TL develo$ments0
2.5 'ow m"ch does the $i%&iew Direct Discovery !eat"re cost? The -li).iew Direct Discovery feature is offered free of charge with -li).iew !!080 2.0 Are there any *rere("isites to "*grade to $i%&iew 11.2? No/ there are not any $rere,uisites to u$grade to -li).iew !!080 &owever/ it is advised to $ursue the u$grade best $ractices0 2.1 2s this a deveo*er ena+ed !eat"re? :es0 -li).iew develo$er should setu$ the Direct Discovery table on the -li).iew a$$lication load scri$t to allow the business users to ,uery the desired data source0
2.3 'ow does $i%&iew Direct Discovery wor%? %ithin the scri$t editor a new synta( is introduced to connect to data in direct discovery form0 Traditionally the following synta( is re,uired to load data from a database table:
To invo)e the direct discovery method/ the )eywords ;S-L SL+T< are re$laced with ;D#R+T -=R:< and to distinguish between data items to be loaded into memory and those which 4ust have metadata loaded the D#>NS#5N?>AS=R?DTA#L )eywords are used as shown below:
ODBC CONNECT TO AdWorks;
DIRECT QUERY DIMENSION CustomerID, SalesPersonID, SalesOrderID, OrderDate, NATIVE('month([OrderDate])') as OrderMonth, NATIVE(Year([OrderDate])') as OrderYear MEASURE SubTotal, TaxAmt, TotalDue DETAIL
6
DueDate, ShipDate, AccountNumber, CreditCardApprovalCode, rowguid, ModifiedDate FROM AdventureWorks.Sales.SalesOrderHeader;
#n the e(am$le above/ the source data table ;Sales5rder&eader< has !@ fields/ with the use of ;D#R+T -=R:< and ;D#>NS#5N< )eywords/ only columns +ustomer#D/ Sales2erson#D/Sales5rder#D and 5rderDate are loaded into memory as symbol tables0 5ther columns following the ;>AS=R< and DTA#L )eywords e(ist in the source data table within the database and they are not $art of the in memory data model0 3oth measure and detail fields are fields that -li).iew is aware of on a ;meta level<0 The actual data of measure?detail fields resides only in the database but the fields may be used in -li).iew e($ressions0 >ore information on the measure?detail fields are $rovided in the ne(t section0
2lease note that $receding load cannot be used with Direct Discovery as it only relates to the data that is loaded in memory0
2.4 What is a 5D26-NS27N8 !ied? A Dimension field can be thought of as a regular in memory field/ all of the data and metadata is loaded into -li).iew and the field is available to be used for associating the Direct table to other in memory tables0 All dimension fields are defined after the D#>NS#5N )eyword within the scri$t and $recedes the measures0
2.19 What is a 56-AS:R-8 !ied? A measure field is a field that -li).iew is aware of on a ;meta level<0 The actual data of a measure field resides only in the database during the reload $rocess and is retrieved on an ad hoc basis driven by the -li).iew front end chart e($ressions0 The idea is that a measure field will be treated as any other field when the user wor)s with e($ressions in -li).iew0 The reason to introduce the conce$t of measure fields is for -li).iew to decide if an aggregation should be done on the database instead of being $rocessed by -li).iew itself0 Aor e(am$le/ if the e($ression +ountBAddressC is used in a -li).iew chart and Address is a measure field/ -li).iew will let the database do the count Bobviously since -li).iew doesn6t have any data for the Address fieldC0 To $ut it in another way1 measure fields will add su$$ort for aggregating in the database while using the same e($ression synta( -li).iew already has0 The result is that the usage of the direct discovery fields from the database will be trans$arent for the user0
7
2.11 'ow are the meas"re !ieds de!ined? =sing the )eyword ;>AS=R<0 5nce this )eyword is used/ -li).iew will only load the fields listed as >AS=R in addition to the fields listed after the D#>NS#5N )eyword0 This scenario can be used by the develo$er in the cases where there are fields on the database table that should be not used by the users on the -li).iew a$$lication0
2.12 2s it *ossi+e to "se so"rce Data+ase syntax in the $i%&iew scri*t? :es0 3y using the new e($ression synta(/ NAT#.BD6C/ it is $ossible to run S-L synta( from a -li).iew scri$t however N7; through a -li).iew chart0 2lease note that the S-L synta( should only be written against the source table defined as Direct Discovery table on the load scri$t0
NATIVE('cast(MktID as varchar(50))') as DetailID e0g0
2.13 Where are the data aggregations done? Aor the -li).iew charts with Direct Discovery fields/ aggregations are done on the database0 #f the chart also has in*memory dimensions/ a second level aggregation is done on the chart level once the database level aggregations are done0
2.1, What is the ex*ected *er!ormance o! the $i%&iew a**ication that "ses Direct Discovery? The $erformance of the Direct Discovery feature fundamentally reflects the $erformance of the underlying system as the feature ,ueries an e(ternal system from -li).iew0 2lease note that/ the in* memory $rocessing will always be faster than in*database $rocessing0 #t is $ossible to use standard database and ,uery tuning best $ractices for this feature0 All of the $erformance tuning should be done on the source database0 Direct Discovery feature does not $rovide any su$$ort for ,uery $erformance tuning from the -li).iew a$$lication0 To im$rove the overall user e($erience/ -li).iew6s caching is used0 -li).iew will store selection states of ,ueries in memory0 As the same ty$es of selections are made/ -li).iew will leverage the ,uery from the cache0 These cached result sets are shared across users0 2lease note that/ when the same ty$es of selections are done/ -li).iew will not ,uery the source data and will leverage the cached result set0 A set statement can be used on the load scri$t to set a caching limit on the Direct Discovery ,uery results Bin secondsC0 5nce this time limit hits/ -li).iew Server will clear the cache for the Direct Discovery ,uery results that were generated for the $revious selections0
8
SET DirectCacheSeconds= 15;
%hen this ha$$ens -li).iew will ,uery the source data for the selections and will create the cache again for the designated time limit0
2.15 What is Dri to Detais? Drill to details is a new ca$ability introduced in !!08 SR5 and allows ;detail< records to be dis$layed in a -li).iew table bo( only0 A detail record is one to which no aggregation is a$$lied and effectively 4ust generates a select statement from -li).iew and renders the a$$ro$riate values0 To cater for the $otential large volume of data which could be rendered an additional Set variable is introduced:
SET DirectTableBoxListThreshold = 100000;
The default value is !EEE0
2.10 What are D-;A2< !ieds? As $art of the drill to details functionality additional synta( has been introduced Bthis follows the D#R+T -=R: statementC: DTA#L * These are the fields Bli)e a F+ommentF fieldC that the user may want to dis$lay in a drill*to* details table bo( but that should not be involved in any chart e($ressions
2.11 What ty*e o! data+ase connection is re("ired to "se $i%&iew Direct Discovery? -li).iew Direct Discovery can only be used against S-L com$liant data sources0 The following data sources are su$$orted: 5D3+?5LD3 data sources * 5D3+?5LD3 sources are su$$orted/ including S-L Server/ Teradata and 5racle0 +ustom connectors which su$$ort S-L G SA2 S-L +onnector/ +ustom -.H connectors for S-L com$liant data stores0 SA2 &ANA via 5D3+ / +loudera?>a$R?&orton %or)s &adoo$/ &2 .ertica via 5D3+0
2.13 Are +oth the 32.+it and 0,.+it connections s"**orted with this !eat"re? :es0
9
2.14 Which $i%&iew aggregation !"nctions can +e "sed with this new !eat"re? As -li).iew Direct Discovery generates S-L as its base code not all -li).iew functionality will be com$atible with the feature0 The functions that are su$$orted with this initial release are1 Sum/ Avg/ +ount/ >in/ >a(0
2.29 'ow is the data+ase connection sec"rity handed? All of the users using the -li).iew a$$lication with the Direct Discovery ca$ability will be using the same connection0 %ith this initial release/ authentication $ass through or credentials $er user is not su$$orted0
2.21 2s it *ossi+e to do asynchrono"s/ *arae cas to the data+ase? :es0 3y using the connection $ooling ca$ability/ it is $ossible to e(ecute multi$le calls to the database0 The load scri$t synta( to setu$ the $ooling ca$ability is as follows:
SET DirectConnectionMax=10; The default setting is !0
2.22 2s it *ossi+e to o*timi)e the Direct Discovery ("eries !rom the $i%&iew a**ication? All of the $erformance tuning should be done on the source database0 -li).iew Direct Discovery feature does not $rovide any su$$ort for ,uery $erformance tuning from the -li).iew a$$lication0
2.23 Are a o! the $i%&iew !eat"res com*ati+e with this !eat"re? Due to the interactive and S-L synta( s$ecific nature of the Direct Discovery a$$roaches/ the following -li).iew features are not su$$orted1 Set Analysis +alculated dimensions Direct Discovery >easure and Detail fields are not su$$orted on 'lobal Search Section access and data reduction Loo$ and Reduce
10
Synthetic )eys on the Direct Discovery table Table naming in scri$t does not a$$ly to the Direct table 2.2, Which data ty*es are s"**orted? All data ty$es are su$$orted/ however there maybe cases where s$ecific data formats need to be defined to -li).iew in order to send the correct literal to the source database0 This can be done on the load scri$t by using a series of ST statements e(am$les are shown below:
Set Statement
Description
DirectDateFormat +ontrols the format of the date literals sent to the database
DirectTimestampFormat
+ontrols the format of the datetime literals sent to the database DirectTimeFormat
+ontrols the format of the time literals sent to the database DirectUnicodeStrings +ontrols whether string literals are $re$ended with the ANS# standard wide character mar)er N0 Not all databases su$$ort this0 Set to TRUE to enable1 any other setting disables0
DirectMoneyFormat +ontrols the format of the money literals sent to the database/ This is not a dis$lay format/ so it should not include currency symbols or thousands se$arators/ default D70EEEE6
DirectMoneyDecimalSep +ontrols the decimal se$arator used in money literals sent to the database0 Note that this character must match the character used in Direct>oneyAormat0 Default is D06
DirectStringQuoteChar +ontrols the character to be used to ,uote the strings in a generated ,uery0 The default is a single ,uote0
DirectIdentifierQuoteChar +ontrols the ,uoting of identifiers in a generated ,uery0 This can be set to either one character Bsuch as a double ,uoteC or two Bsuch as the $air FIJFC0 The default is a double ,uote0
DirectCacheSeconds +ontrols the amount of time a ,uery result set from an
11
a$$lication is stored in memory cache/ default setting is K@EE seconds0
DirectTableBoxListThreshold +ontrols the amount of drill to detail data available to be shown in a table bo(/ the default is !EEE rows0
DirectDistinctSupport +hanges the default synta( of SL+T D#ST#N+T to use 'R5=2 3: for sources which do not su$$ort D#ST#N+T
DirectIdentifierQuoteStyle Aor non*standard ANS# ,uoting anything other than DANS#6 in the statement will change the synta(
2.25 What ty*e o! S$< syntax can +e "sed on the oad scri*t to set"* the Direct Discovery? The first release of -li).iew Direct Discovery only su$$orts the use of one in*database table Bcan be a viewC along with e(ternal data sources Bloaded in memoryC0
2.20 What are the ogging ca*a+iities with the Direct Discovery !eat"re? 3ehind the scenes a direct S-L statement is $assed to the underlying data source as such the statement $assed can be viewed through the trace files of the underlying connection0 An e(am$le of this can be found in the 5D3+ Data Source Administrator in control $anel ;Start Tracing Now< button0 The subse,uent trace file details S-L statements generated through the user selections and interactions