Microsoft SQL Metadata to Catalog

Data Source Name Microsoft SQL

Use Case BI & Analytics

Data Lakes

Target Audience Developer (Integration Engineer)

Who can set it up Developer (Integration Engineer)

Target Business Any Business Function

Functions

Target Industries Any Industry

License Collibra Connect or Collibra Catalog

Requirements

Dependencies Collibra DGC 5.x

Marketplace URL https://marketplace.collibra.com/listings/microsoft-sql-metadata-to-catalog/

Developer Catalog Team

Short Description This solution approach will leverage the Catalog SQL Server jdbc connector to ingest metadata directly from the
Microsoft SQL Server storage tier.

Overview
Functional Design
Installation
Configuration
Usage
Release History
Troubleshooting

Overview
Microsoft SQL Server is a relational database management system. As a database server, it is a software product with the primary function of
storing and retrieving data as requested by other software applications—which may run either on the same computer or on another computer
across a network. This solution approach will leverage the Catalog SQL Server jdbc connector to ingest metadata directly from the Microsoft SQL
Server storage tier.

More about Microsoft SQL:

MS SQL doc

Functional Design
Asset types used for integration:

Asset Description
type

Database A collection of data that is systematically organised or structured in order to make it easy to create, update and query the
information. Examples: Ora_DGC_V45, SalesDB2020

Schema An organised structure described in a formal language supported by implementing technology that defines the objects in the
technology assets (Table and columns in a relational database, fields in a file). E.g. CRM_001_PRD, HDP_CNT_CLD

Table An implementation of Data Entities in columns and rows, in a given database system. It is the basic structure of a relational
database. Examples: Account_tbl, CUST_ADDR

Column An atomic unit of data that can be stored in a database table. Examples: FST_NM, EMPID
More information is here: JDBC integration metadata model.xlsx.zip

Installation
To perform data source ingestion, JDBC driver should be added to instance.

JDBC jar-file can be downloaded from: MSSQL JDBC Driver

Information how to do this is provided in Documentation or in the section "Configuration" below:

"Configure new data source driver"

"Add and configure JDBC driver for data source registration"
"Install a new JDBC driver version"
"Update a JDBC driver"
"JDBC connection details"

Configuration
For Microsoft SQL Data source JDBC-driver is added to the product by default.
But there is a possibility to add other JDBC-drivers also:

1. Download JDBC-driver (see source in section "Installation")

2. Open wizard and start to Register Data Source (Catalog -> Global "Create" button)

3. Select "SQL Server"

3.

4. Add schema name (mandatory), description (optional) and select owner (by default, current loged-in user)

5. Click on drop-down under the title JDBC driver version" and select "manage drivers" (more information can be found in "Add and
configure JDBC driver for data source registration")
5.

6. In new opened window add jar-file

7. After jar-file is added, JDBC driver should be configured (more information can be found "in "JDBC connection details")
 8. When configuration is finished you can proceed data source ingestion process.

Usage
1. Open wizard and select "Register Data Source" -> "SQL Server" (more information is in "Registering a data source")
2. Add information about Schema
3. Select Job Server and add needed credentials

4. Now there are possibilities to choose and perform "Store Data Profiling", "Detect advanced data type", "Store Sample Data" or exclude
from the registration process some of the tables.
 4.

5. As a final step you can use with the result of ingestion for your purposes.

Release History
v 5.0 -- Initial release. List main features:
10/18/2016
Ingest data
Assemble data sets
Enrich data sets
Shop for data

v 5.1 -- 06/16/ Data profiling on registered data sources

2017 Job server to run the data profiling jobs

v 5.2 -- 09/22/ Foreign Key ingestion

2017

v 5.3 -- 01/19/ Inline editing of advanced data types

2018 Bulk delete of advanced data types
New chart icon for a bigger preview
Register data source can now be started from any asset page of the asset types Schema, Data Set, Table, Column
Updated register data source flow

v 5.4 -- 06/08/ The Collibra Catalog Home page is designed to help you quickly and easily find Catalog-related assets
2018

v 5.5 -- 09/07/ UX improvements

2018

v 5.6 -- 01/25/ New layout of Catalog pages to improve the user experience (Data sets, Tables, Schemas)
2019 Support for multiple Jobservers: install a Jobserver close to your data sources, even when they are in different network
silos, to increase the performance
An API to store profiling information in Catalog, even if Catalog doesn't natively support the data source.

Troubleshooting

Feedback & questions are welcome!

If you have any comments or questions regarding this listing, please feel free to post your comments or questions on the Marketplace
Users & Developers group: https://community.collibra.com/groups/marketplace/ or on our slack channel #developers