Building Your ETL

Framework with BIML

Meagan Longoria
KCDC 2015
  Slides are on my blog
 Feel free to share questions and comments throughout the
session

Begin at the
Beginning
 Meagan Longoria
Business Intelligence Consultant at
BlueGranite
Organizer of SQL Saturday Kansas City
(Oct 3, 2015)

Who Are You? Blog: http://datasavvy.wordpress.com/

Twitter: @mmarie
LinkedIn:
www.linkedin.com/in/meaganlongoria/
8 years in BI
7 years working with SSIS
  Business Intelligence Markup Language

 Domain specific language for describing business

intelligence objects
 XML that you can write to:
What on Earth  Build packages faster
 Ensure consistency
is BIML?
 Free – comes with BIDS Helper
 Also available in Mist
  Allows you to extend BIML with C# or VB.NET
 “Like ASP for BIML”
 You can use BIMLScript to:
 Import database table schemas (quickly create/update your dev
environment to look like prod)
 Replace static values with expressions
BIMLScript  Include text from another BIML file or text file
 Turn tedious, repetitive work into reusable scripts

 BIML is still useful without BIMLScript, but much more powerful

with it
  2008 - BIML was born
 2008 - Varigence was founded by Scott Currie
 2009 - BIMLScript was created
A Little  2011 - BIML compiler added to BIDS Helper
Background
 BIML describes
 SSIS packages, databases, schemas, tables, columns
 SSAS cubes, facts, dimensions (Mist only)
 BIMLCompiler
How It Works BIML/BIMLScript (Mist/BIDS DTSX packages
Helper)
  All generated artifacts appear to be hand built
 Packages can be deployed and run on unmodified SQL Server (no
need to install anything on the server running the packages)
 Free with BIDS Helper
 Benefits:
BIML is  Reduce amount of time it takes to develop an SSIS project
Awesome 

Help you recover from Drag-and-drop-itis
Ensure a consistent design pattern
 Stop solving the same problems over and over and move on to
something new and interesting
  https://bidshelper.codeplex.com/
 Choose the correct install for your version of SSDT/SQL Server
 Current version is 1.7
BIDS Helper  Now properly supports SSIS 2014/Visual Studio 2013
 Several BIML Updates, some breaking changes
  IDE for authoring BIML code
 Build via GUI or type code
 Includes text editors with syntax highlighting, intellisense and
quick-info displays, source control, and multi-monitor support
Mist  Import existing databases and SSIS packages
 Free 14-day trial: reverse engineer 5 packages
 Perpetual or subscription licenses available
 <Biml xmlns="http://schemas.varigence.com/biml.xsd">
BIML Basics <Packages>
<Package Name="MyBlankPackage">

</Package>
</Packages>
</Biml>

Demo
 Created a blank package

Created connections

Created a staging package

My Learning Reverse engineer packages using Mist

Process
Use the BIMLScript include

Create BIML library for common patterns (DB table

staging, flat file import, fact , Type 1 SCD, Type 2 SCD)
  Just BIML, no script
 Team members with little SSIS experience
 Trying to ensure a consistent design pattern
 Needed to finish the project quickly
 SQL Server 2012, project deployment model
 12 dims, 4 facts, 18 staging tables, 4 connections
My First BIML
Project  Used template BIML to generate packages
 Saved pattern in BIML using AdventureWorks
 Sent instructions to junior members on which items to change
(connection, source query, destination table, etc.)
 Had them generate the package without saving the BIML changes
Boost It With  Include

BIMLScript  CallBIMLScript

Demo
  3 schemas: Audit, Stage, Prod (whatever the project/subject area
is called, ex: HR)
 Audit schema contains two tables:
 PackageControl: Package execution details for easy querying and
restartability
 PackageDependency: lists packages in groups and orders them for
execution
My SSIS  Staging tables include all columns from source table for thin
Framework tables, select columns for wide tables, + audit fields
 Views used to transform data and do lookups for dims and facts
 Master packages for staging, dims, facts use package dependency
for order and concurrency and package control for restartability

Demo
  BIML files are included with each new SSIS project for source
control
 You decide whether you source control BIML, SSIS packages, or
both. The right answer depends on your work processes.
My BIML  My recommendation: Although you could describe tables in BIML,
it is best to continue use of database projects to contain your table
Library schemas, views, stored procedures
 Troubleshooting and updates tend to occur in the database rather
than the package once your design patterns are set.
  Get ready to bang your head a few times. It’s fine. You won’t break
anything.

 Learn to abstract package definitions and use

Lessons properties/parameters in BIML script to fully automate your
package development.
Learned
 BIML makes SSIS more accessible for .NET devs who occasionally
dabble in SSIS.
  Varigence BIML forums:
https://www.varigence.com/Forums?forumName=Biml
 BIMLScript.com http://bimlscript.com/Develop/Resources
 Stairway to BIML:
http://www.sqlservercentral.com/articles/BIML/100552/
 BIDS Helper documentation:
Resources for https://bidshelper.codeplex.com/documentation
Further  http://geekswithblogs.net/darrengosbell/archive/2015/04/24/bid
s-helper-1.7.0-released.aspx
Learning
 https://varigence.com/Documentation/Samples/Biml/
 http://billfellows.blogspot.com/
 Feel free to contact me with questions or feedback.

Meagan Longoria
Final Blog: datasavvy.wordpress.com
Questions and Twitter: @mmarie
Comments LinkedIn: www.linkedin.com/in/meaganlongoria/
Company Website: http://www.blue-granite.com/