SQL Server Cert ( 70-764 )

Notes during review

- be familiar with the different 'editions' of SQL server:

https://www.microsoft.com/en-us/sql-server/sql-server-2017-editions
- Express
- 4 core limit , 1410MB max buffer limit , 352MB columnstore limit , 352MB max optimized
data limit , 10GB database size limit
- Developer
- 4 core limit
- Standard
- 24 core limit ,
- Enterprise
- unlimited cores, OS limit is max buffer limit , OS limit is columnstore limit , OS limit is the max memory
optimized data per database , max database size 524 PB

- Database Structures
- two categories:

1) OLTP - online transaction processing

- this is your day to day business operational data
- as it gets bigger over time.. reading data from it becomes less efficient

2) OLAP - online analytical processing ( data warehouse / data mart )

- very fast at retrieving data
- not fast as edit's, inserts, or updates

- Transact - SQL or T-SQL : is a flavor of the standard Structured Query Language

- T-SQL , was created by Microsoft and Sybase

- Installing SQL Server

- Verifying SQL Server Installation

 - Tools
- SQL Server Installation Center
--> access to "Installed SQL Server features discovery report"
creates a .html file report
file:///C:/Program%20Files/Microsoft%20SQL%20Server/130/Setup
%20Bootstrap/Log/20200214_154220/SqlDiscoveryReport.htm

- SQL Server Security

4 main concepts:

1) Authentication
- Who?
- Validates the identity of the user
- Can be performed based on three things:
- something the user knows (password or shared secret)
- something the user owns (card, badge, etc... )
- physical characteristics (fingerprint, etc.. )

2) Authorization
- What?
- Determines what resources the authenticated user can access and what actions they
can perform

3) Auditing
- Also called change tracking
- Creates and maintains a record of user actions

4) Encryption
- The process of storing data in scrambled form, making it unreadable to unauthorized
users
- Used in different forms and locations within the SQL Server database platform

- SQL Server security model is implemented at three levels

1) Principals
 - Windows user accounts
- SQL Server logins
- Windows groups
- SQL Server roles
2) Securables
- Any resource or object whose access is regulated by the system based on
authentication
3) Permissions
- Properties that are set on securables that are used by the system to
determine what actions an authorized principle can perform

- Server Level Security

- a single server can host multiple databases... thus:
- SQL Server security is implemented at two levels
- Server
- Database

1) Users first authenticate to the SQL Server

- two methods for authenticating
1) Windows authentication mode
2) SQL Server and Windows authentication mode ( mixed mode )
2) Next they must authenticate into the database(s) which they have been granted
permissions

- Windows Authentication mode:

- Also called a trusted connection
- SQL Server trusts the users authentication at the operating system level
-> their Windows user account
- This is the most secure method
-> not an option for users who do not have a Windows user account

- SQL Server and Windows authentication mode

- Mixed mode
- Login accounts can be created and maintained in the SQL Server security system
- SQL Server will also trust the OS login
 - This mode is useful when access to SQL server is needed by a user who does not have a
user account in the Windows domain

** the authentication mode is set during installation **

- it can be changed at any time... but a SQL Server restart is required

Restarting SQL Server

- Some configuration changes require the SQL Server to be restarted
- ways to restart:
1) in SSMS right click on the SQL Server, on pop-up menu choose "Stop" , "Restart" ,
"Start" , "Resume", or "Pause"
2) in Windows "Services" , right click on SQL Server (MSSQLSERVER) and choose
"Restart", "Stop" , "Start" , "Resume", or "Pause"

Creating Logins
- NOTE :
- Logins : are accounts that connect to the SQL Server
- Users : are accounts that connect to a specific database
- Creating logins at the server level can be done two ways:
1) in SSMS
2) SQL script "CREATE USER" command
CREATE LOGIN [DESKTOP-R4UOPBJ\newUser] FROM WINDOWS

- Database Level Security

- User account is how one gains access to a specific database
- 2 Types of Database user accounts
1) Accounts based on logins at the server level
- SQL Server logins and Windows accounts that are stored in the master database at the
server level
- For users who need access to multiple databases on the server
2) Contained database user
- User account is not associated with a master DB login

- Database User Account Types

 - provide access to a specific database
- User Types:
- SQL user with login
- SQL user without login
- User mapped to a certificate
- User mapped to an asymmetric key
- Windows user
- 2 categories:
1- Users that are based on a database login ( stored in Master database )
- user account is based on a login acocunt that is based on a Windows Active
Directory user account or group
( mixed-mode authentication )
- user account is base on a SQL Server Login
( SQL Server authentication )
2- Users that are not based on the master database, but authenticat to a individual
database (commonaly refered to as "authenticate at the database")
- user account is based on a Windows user or group that has no SQL server
login
- Contained database user with password
- User without a login:
- cannot login but can be granted permissions
- used for impersonation
- User based on a certificate
- cannot login but can be granted permissions and can sign modules
- User based on a asymmetric key
- cannot login but can be grantedd permisions and can sign modules

- Cross-Server Authentication
- commonly enterprise data is stored in multiple databases across multiple SQL Servers
- a linked server configuration enables SQL server to execute commands against OLEDB data
sources on remote servers
- Once a linked server has been configured, the power of the four part object naming in T-SQL becomes
apparent:
- SELECT * FROM <server>.<database>.<schema>.<object>
-example:
 SELECT * from Srv2.DB1.Sales.Customer

- Linked Servers offer a number of advantages:

- Remote server access
- The ability to issue distributed queries, updates, commands and transactions on
heterogeneous data sources across the enterpirse
- the ability to address multiple data sources similarly
- Linked servers are created using SSMS or the "sp_addlinkedserver" system stored
procedure
- see 'create linked servers.pdf' in the Working Filed folder
- The required configuration details vary depending on the data source:
- different remote data sources will require various types and numbers of parameters
- some only provide read-only access
- Linked Servers are not super popular in the real world due to the increased security risk

Partially Contained Databases

- SQL server has historically maintained a strong relationship between each database and the
SQL Server that hosts them
- this can create challenges when a database needs to be moved due to
administrative, operational or availability reasons
- the database does not contain all the necessary settings and metadata to
function on it's own
- moving a database causes important information to become unavailable
Example:
- logins that are stored in the master database are no longer available
- these must be manually moved

** as of now, SQL Server only supports non-contained(default) and partially contained databases,
someday the goal is for it to support Fully Contained databases that can be easily moved between
servers **
(partially contained)
- databases can be created as partially contained or migrated to a partially contained
database
- much of the metadata that describes a database is maintained in the database
- user authentication can be performed by the database, reducing the database dependency on
the logins of the instance of SQL Server
Dynamic Data Masking
- Database tables often contain various forms of sensitive data:
- SSN
- Account #
- CC numbers
- etc..

- Dynamic data masking provides a efficient solution to protecting sensitive data:

- it masks data to non-privileged users
- it can mask a predetermined amount of the sensitive data
- all masking is done at the database level without modifying the actual data
- masking is applied automatically to query results based on dynamic data
masking rules

- DDM functions based on rules that are defined on a table column using T-SQL
- 4 Types of data masks can be applied:
- Default ( masks data based on it's data type )
- binary, varbinary, numeric, image, all masked with 0s (zeros)
- date/time is masked with 01.01.1900 00:00:00:0000000
- string data is masked with Xs
- Email ( does this: aXXXXX@XXXXXX.com )
- only 1st character and suffix remain
- Random ( masks the original value with a random value within a specific
range )
- Custom String ( exposes the 1st & Last letters and adds a custom padded
string in the middle )
- DDM is a Column level functionality
example: this applies a mask to the SSN column
Alter Table SensitiveData

Alter Column SSN Add Masked

With (Function='partial(0,"XXX-XX-",4)')

- Understanding Roles
- All versions of SQL Server use role-based security
 - Users are groupe into Rolls
- Permissions are assigned to the roles instead of individual users

- Role : is a container that can contain other principals

- Permissions can be assigned to the role
- Members of the role inherit the role's permissions
( similar to groups in the Windows operating system )

- SQL Server provides pre-defined roles at the server and database level:
- Fixed server roles have a pre-determined set of permissions
- the permissions on the roles cannot be changed
- Fixed database roles are provided for each individual database
- the permissions on the roles cannot be changed
- Fixed Roles have two main characteristics
-1 Greatly simplify permissions management
-2 Greatly increase the granting of too many permissions
- you should never grant a user permission to perform actions that are
not required to perform their job
- WATCH for this scenario on the exam !

- You can create your own user-defined roles

- assign any combination of permissions desired

- Fixed Server Roles ( provided by SQL )

- are designed to apply administrative permissions to users
- provided for convenience and backward compatibility
- Microsoft recommends that you create user-defined roles with more
specific permissions whenever possible
- 9 provided fixed Server roles:
- Sysadmin : members can perform any activity in the server
- Serveradmin : members can change server-wide configuration options and
shut down the server
- Securityadmin : members of this role can manage logins and their
properties
grant, deny and revoke server-level and database-level
permisions if they have access to the database
 reset passwords for SQL Server logins
- Processadmin : members can end processes that are running in an instance
of SQL Server
- Setupadmin : members can add and remove linked servers by using
Transact-SQL statements
Sysadmin is required when using SSMS
- Bulkadmin : members can run the BULK INSERT statement
- Diskadmin : members can manage disk files
- Dbcreator : members can create, alter, drop and restore any database
- Public : (be careful with this one )
Every SQL Server login belongs to the public server role
When a server principle has not been granted or denied
specific permissions on a securable object, the user inherits
the permissions granted to public on that object
Only assign public permissions on any object when you want
the object to be available to all users

- Fixed Database roles are designed to apply database-level administrative

permissions to users:
- Each of the roles are found in each individual database on a SQL Server
- db_owner : members can perform all configuration and
maintenance activities on the database
can also drop the database in SQL Server
- db_accessadmin : members can add or remove database access for
Windows logins/groups, and SQL Server logins
- db_datareader : members can read all data from all user tables
- db_datawriter : members can add, delete, or change data in all user
tables
- db_ddladmin : members can run any Data Definition Language (DDL)
command in a database ( CREATE, ALTER, etc.. )
- db_securityadmin : members can modify role membership and manage
permissions
- db_backupoperator : members can backup the database
- db_denydatareader : members cannot read any data in the user tables
within a database
- db_denydatawriter : members cannot add, modify, or delete any data in the
user tables within a database
- public : every database user is automatically a member and cannot be
removed
any permissions applied to the public role are applied to all users
 only assign public permissions on an object when you want the
object to be available to all users

- User defined roles : custom user-defined roles can be created at the server and
database level using SSMS or T-SQL statements:
[Server Level] - CREATE SERVER ROLE
- DROP SERVER ROLE
- ALTER SERVER ROLE

[Database Level] - CREATE ROLE

- DROP ROLE
- ALTER ROLE

- At the database level, Active Directory groups can be

added to roles
- adding an AD group to a database level role assigns
all role permissions to the AD group
- leveraging the organizational work performed in
creating the AD groups
- multiple AD groups can be added
- removing an AD group removes the granted
permissions from the users in the AD group but does
not affect permissions on other role members

- GRANT, REVOK, and DENY : authorized users can only perform actions that have
been explicitly granted to them:
- to their user account
- to roles or groups asociated with them
: it is very important to understand the implications of
the relationship between GRANT, REVOKE, and DENY
GRANT : provides the ability to perform a specific operation on a object
REVOKE : removes an assigned GRANT permission
does not restrict the user from any permissions that may be
applied elsewhere :
- A group or role the is a member of
DENY : denies permission to a specific object
overwrites all GRANT permissions
the user will not be able to perform the denied permission regardless
 of any granted permissions in other locations
** Be careful when assigning DENY permissions **
A DENY assigned to a group or role will remove the permission for all users
of the group or roll

Schema Level Permissions ( applied at the Database level )

- Schemas provide three very important functions in SQL Server:
1) Separate direct ownership of objects from individual users
( this simplifies ownership issues when managing objects )
2) Container that logically groups tables, procedures, views, etc.. together
3) Securable object ( can be backed up )

Managing access permissions on an object-by-object basis quickly gets complex

- as the database grows, the complexity increases
Schemas provide the ability to greatly simplify database organization and permissions
management
- permissions can be applied at the schema level
- these permissions apply to all objects in the schema

SQL Server Encryption Basics

1) First level of protecting data is the implementation of authentication and authorization

- Authentication : attempts to validate a user’s identity
- Authorization : controls what actions an authenticated user can take
“Authentication/Authorization functionalities can be breached”

2) Another level of protection on the data is Encryption

- ‘Scrambling’ the data using a key or password
- Encryption does not solve access control problems
- It enhances security by limiting data loss
- Encrypted data that is accessed or stolen is useless without the ability to decrypt it

Can Encrypt the data in its two main states:

1) Data at rest
 - data that sitting in storage
2) Data in transit
- data that is traveling over a network
3) Encryption can be utilized in SQL Server t protect three categories of database
a) Connections
b) Data
c) Stored Procedures

4) SQL Server encrypts data using a hierarchical encryption and key management infrastructure
- Each layer encrypts the layer below it
→ Utilizing a combination of certificates, asymmetric keys and symmetric keys
→ Asymmetric and Symmetric keys can be stored outside of SQL Server in an Extensible Key
Management ( EKM ) module for an added layer of key protection

The Dark Side of Encryption

- Encryption appears to be the silver bullet in data protection

→ It is very effective
→ It can be destructive

- Can have two negative effects

→ Performance
→ Data Los

- Encryption will affect performance:

→ the process of encryption and decryption requires CPU recources
→ More encryption will have a bigger performance impact
→ Different encryption implementations in SQL Server will have different performance impacts
→ Monitor for real-world impact

- Encryption is based on the use of keys

→ The encryption keys must be protected
→ They must also be backed up in a separate, safe locations

- If encrypted keys are unavailable, the data cannot be decrypted

→ same result as losing the data

- Key management can evolve into a significant administrative task

Asymmetric vs. Symmetric Encryption

- There are two options when using encryption keys
→ Asymmetric keys
- two keys are used in the encryption / decryption processes
- the keys are algorithmic opposites
- one key Is used to encrypt the database
- the opposite key is used required to decrypt the database
- ( generally considered “more secure” )
→ Symmetric keys
- the same key is used to encrypt and decrypt the data

- The real difference is speed and security

→ Symmetric encryption/decryption is faster
→ Asymmetric encryption is more secure
- The two key types are often used together
→ A Symmetrical key is encrypted by an asymmetric key for more protection during storage
 → The Symmetric key is decrypted when needed and used to encrypt and decrypt database

Encryption Options
- in SQL server 2016 you will often have the option to choose the encryption algorithm

- SQL Server 2016 offers three encryption algorithms

- AES_128
- AES_194
- AES_256

- What is the difference between these?

→ the difference is in the key sizes of the encryption algorithms
→ AES_128 is 128-bit encryption
→ to guess a 128-bit key requires 2 to the 128 th guesses to try to identify each of the
possible 128-bit keys
→ AES 128-bit is a good choice
→ Not using 256-bit will result in questions from clients, supervisors, etc…
→ The Encryption options in SQL server 2016 will provide excellent security until
quantum computing becomes a reality
→ AES_193 is 192 bit encryption

Encrypted Backups
- Backups protect data from loss
→ potentially expose it to theft
- Encrypting the data backup offers an extra level of data protection
- SQL Server 2016 provides the ability to encrypt data backups
Requirements
→ A database master key for the master database
→ A certificate of asymmetric key available on the instance of SQL Server
- Once those objects have been created, we can perform the backup
- Database master key
→ The database master key is a symmetric key
→ Used to protect the private keys of certificates and asymmetric keys that are present I the
 database
→ It can also be used to encrypt data, but it has length limitations that make it less practical for
data

- Creating a Certificate
→ A Certificate is a database-level securable that follows the X.509 standard and supports X.509
V1 fields
→ A Certificate can be created for a database by executing the CREATE CERTIFICATE T-SQL
statement
→ Load a certificate from a file or assembly
→ Generate a key pair and create a self-signed certificate

- Documents provided in the Working Files folder with more information

- Create Certificate.pdf
- Backup Certificate.pdf
- Buck Up a Database Master Key.pdf
- Back Up the Service Master Key.pdf

- -- create the master key

USE master;
Go
Create Master Key Encryption by password = 'TestPasswordForEncryptedBackup06252020';

-- command to check if a master key already exits

--
select * from sys.symmetric_keys

-- create certificate
Use Master
go
Create Certificate TestDBBackupEncryptCert
With Subject = 'TestDB Backup Encryption Certificate';

-- backup database
Backup Database [AdventureWorks2012]
To Disk = N'C:\Program Files\Microsoft SQL
Server\MSSQL13.MSSQLSERVER\MSSQL\Backup\AdventureWorks2012.bak'
with
compression,
encryption
(
Algorithm = AES_128,
Server Certificate=TestDBBackupEncryptCert
),
STATS=15
go
Transparent Data Encryption
- Transparent Data Encryption (TDE) protects data at rest
→ Performs real-time I/O encryption and decryption of the data and log files
→ The data is encrypted before it is written to the disk
→ Decrypted when the data is read from the disk
→ All encryption/decryption is performed transparently in the background

- Implementing TDE
→ Create a master key (DMK), if one doesn’t exist
→ Create or obtain a certificate protected by the master key
→ Create a database encryption key (DEK) and protect it by the certificate
→ Set the database to use encryption using the ALTER DATABASE statement

- A few considerations when using TDE

→ When TDE is implemented, database backups are encrypted
→ Restoring the database requires the availability of the certificate that is protecting the
database encryption key (DEK)
→ You must maintain backups of the server certificates as well as the data
 → TDE will encrypt all files and filegroups in the database
→ If any filegroups are marked read only, the database encryption operation will fail
→ TDE does not increase the size of the encrypted database
Implementing TDE
-- Implementing TDE example

-- first check to see if TDE is already implemented

select [Name], is_encrypted
from master.sys.databases

-- create the master key ( if does not already exist )

Create Master Key Encryption by Password = 'gasduihobHijdgUL697##';

-- check to see what keys already exist

Select * from sys.symmetric_keys

-- check to see what certificates already exist

Select * from sys.certificates

-- Create certificate
Create Certificate MyServerCert
With Subject = 'My DEK Certificate'

-- backup cartificate
Backup Certificate MyServerCert To
File='C:\Program Files\Microsoft SQL
Server\MSSQL13.MSSQLSERVER\MSSQL\Backup\MyServerCert'

-- create the database encryption key

USE AdventureWorks2012;
go
Create Database Encryption Key
With Algorithm = AES_128
Encryption By Server Certificate MyServerCert

Alter Database AdventureWorks2012

Set Encryption On;
Go

Always Encrypted
- Database administrators, and various other personnel, possess two main characteristics:
→ The are an invaluable asset to the enterprise
→ They are the biggest single security threat

- Sensitive data in the database can be accessed by a variety of necessary, high-privileged personnel:
→ Local administrators
→ Cloud database personnel
→ Vendors, clients, etc.
- Data Security has now evolved to identify two distinct classes of personal:
→ Those who own the database
→ Those who manage the data
- SQL Server 2016 introduces Always Encrypted
→ Allows data owners to access encrypted database
→ Does not allow data access to anyone else, including database administrators
→ Never reveals the encryption keys to the SQL Server Database Engine

- How does Always Encrypted work?

→ Data is encrypted and decrypted a the application level
→ An encrypted-enabled driver on the client computer encrypts data before writing it I
into the database
→ The driver decrypts the data when it is retrieved
→ Implemented at the column level

- Always Encrypted is configured for individual database columns that contain sensitive database
→ Specify the column(s) to be encrypted
→ Specify an encryption algorithm to use
→ Specify cryptographic keys to be used

- Always Encrypted uses TWO types of keys:

1) Column Encryption keys
→ used to encrypt data in an encrypted column
2) Column Master Keys
→ used as a key-protecting key
→ encrypts one or more column encryption keys
- Two Types of encryption can be used when configuring Always Encrypted
1) Deterministic encryption
→ Always generates the same encrypted value for any given plain text value
→ Allows the database engine to perform lookups, equality joins, grouping and indexing on
encrypted columns
→ May allow unauthorized users to guess data values
( especially values such as true/false, north/south/east/west, etc… )
2) Randomized encryption
→ Encrypts the data in less predictable manner
→ More secure, but prevents searching, grouping, indexing, and joining on encrypted columns
- see “Always Encrypted.pdf” file in working files folder

Implementing Always Encrypted

- in SSMS
→ right click on the AdventureWorks2012 database
→ then choose the “Tasks” option
→ then choose “Encrypt Columns”
→ this starts the “Always Encrypted” Wizard
→ click “next”
→ then click on “Column Selection”
→ choose the table and column you want to encrypt
→ click the check box of the column , it’s encryption type, and key
→ click next
→ on the Master Key Config screen choose the “Auto generate column master key”
option
→ then select the “Windows certificate store” option
→ then “Current User” for master key soure and click “Next”
→ On “Run Settings” screen you can clisk “Proceed to finish Now” and click “Next”
Index Management Basics

- Indexes have a major impact on how data is stored, accessed, and managed in SQL Server
- Determine the physical storage order of data
- Allow the data to be much more efficiently managed
- Create performance issues

- An index on a database table provides the same efficiencies that an index in the back of a book
provides
- The book is sorted by page number
- The index lists subjects or words in alphabetical order and provides a ‘pointer’ to the page
number where the data can be located

- SQL Server 2016 provided two types of indexes

1) Clustered
- a clustered index causes the rows in a table to be physically stored in a certain order
- similar to the page numbers in a book
- based on the column(s) that make up the clustered index
- there can only be one clustered index on a table
- most tables should have clustered index
- simple rule for choosing the clustered index column(s):
- the column(s) that are returned in the most often used queries ( such as last and first name )
- the maximum key size for a clustered index is 900 bytes
( so the total size of the column(s) data types in the clustered index cannot exceed this size )

2) Nonclustered
- a table that does not have a clustered index is called a heap
- the data is not stored in any particular order ( inserts are very fast )
- a separate index structure that is independent of the actual storage order of the data rows
- similar to the index in a book
- a ‘what-if’ way of viewing the column(s)
- up to 999 nonclustered indexes can be created on a table
- the maximum key size for nonclustered index in SQL Server 2016 is 1700 bytes

2) Nonclustered (cont..)
- ** Nonclustered indexes can slow down data modifications
- the indexed information is created in an index structure in the order of the index
- this range(index) can be very efficiently searched
- pointers are used to locate the actual data rows
- this can slow down data modifications !!
- these are great for searches.. but not for data updates
- updates now cause the pointers to be reorganized
- the more nonclustered indexes are added, the more inefficient modifications become

Over time as data is modified, deleted, and inserted the indexes become disorganized in the way they
are stored.
- this disorganization is called index fragmentation

One of the many responsibilities of a SQL database administrator is to manage the indexes that exist on
the tables
- Repair fragmentation
- this can be part of an automated database maintenance plan
- Identify underutilized indexes
- modify or remove them

Identifying Fragmentation
- as data is modified in a table, index structures become fragmented
- the logical order of data in the index differs from the physical order in the data file
- results in page order scrambling and free space in the data, requiring higher I/O when scanning
the index

- indexes need to be managed by performing defragmentation when needed

- indexes are managed in two ways:
1) automatic maintenance by the database engine
2) manual intervention
-

Automatic Index maintenance by the database engine

- SQL Server database engine performs automatic maintenance when inserts, updates, or
deletes occur
- This maintenance lead to index fragmentation over time
- the logical order of data in the index differs from the physical order in the data file
- results in page order scrambling and free space in the data, requiring higher I/O when
scanning the index

Manual Index intervention

- repairing index fragmentation is accomplished by periodically reorganizing or rebuilding the
indexes
- the need to reorganize or rebuild is based on the amount of fragmentation that exists
- the system function sys.dm_db_index_physical_stats returns size and fragmentation
information for tables and views

Sys.dm_db_index_physical_stats function provides:

- avg_fragmentation_in_percent ( Average percent of incorrect pages in the index )
- fragment_count ( Number of fragments in index )
- avg_fragment_size_in_pages ( Average number of pages in one fragment in an index )

Rebuild or reorganize based on the AVG_FRAGMENTATION_IN_PERCENT column…

- less than 10% : no defragmentation is required
- 10-30% : the index should be reorganized
- more than 30% : the index should be rebuilt

Repairing Fragmentation
- index fragmentation can be repaired by either rebuilding or reorganizing the indexes
- indexes can be rebuilt or reorganized in two ways:
1) Graphical tools in SSMS
2) T-SQL

-- Repairing Index Fragmentation Example

-- Execute this query
-- Uses sys.dm_db_index_physical_stats
-- Change the database name and table name as needed
SELECT a.index_id, name, avg_fragmentation_in_percent
FROM sys.dm_db_index_physical_stats (DB_ID(N'AdventureWorks2012'),
OBJECT_ID(N'HumanResources.Employee'), NULL, NULL, NULL) AS a
JOIN sys.indexes AS b ON a.object_id = b.object_id AND a.index_id = b.index_id;

-- To reorganize an index
ALTER INDEX IX_Employee_OrganizationalLevel_OrganizationalNode ON HumanResources.Employee
REORGANIZE;

-- To reorganize all indexes on a table

ALTER INDEX ALL ON HumanResources.Employee
REORGANIZE;

-- To rebuild an index
ALTER INDEX PK_Employee_BusinessEntityID ON HumanResources.Employee
REBUILD;

Database Engine Tuning Advisor ( DTA )

- The Database Engine Tuning Advisor (DTA) is a built-in tool
- Examines how queries are processed in the databases you specify
- Recommends modifications that may improve query processing performance
- Database structures including indexes, indexed views, and partitioning

- DTA provides two user interfaces:

- GUI
- dta command prompt utility

- The DTA utilizes data that can be provided in three ways

1) run SQL Server Profiler to capture a trace file or trace table of actual database activity
 2) Manually execute T-SQL commands or scripts in the Query Editor while SQL Server Profiler is
capturing data
3) If the Query Store feature is enabled, the DTA can use the data it has collected
( no need to collect data with SQL Server Profiler )
- DTA can also be opened from the SQL Server Profiler

Backup Plans

- The need to perform a database restore is not a probability:

- at some point, it becomes a certainty
- the efficiency and effectivenes of the restore is not as certain
- Tired of your current job as a database administrator?
- ignore the importance of a database backup plan

- The secret to a successful, efficient recovery after an even involves two main steps:
1) Create a backup plan
2) Test the backup plan

- The best way to coalesce a backup plan is to ask questions:

- How much data loss is acceptable?
- How often does data in the database change?
- What is an acceptable recovery time?
- How big is the database maintenance window?
- How large is the database?
- What is the budget for database backup and recovery?

- The answers to these questions will solidify the reality and need for a backup plan:
1) How much data loss is acceptable?
2) What is the budget for database backup and recovery?

- The three most important aspects of a good backup plan

1) Assignment of backup responsibilities
2) Step-by-Step directions for backup and recovery
3) Testing the entire process/solution
Backup Types
- There are six types of backups that can be performed in SQL Server 2016
1) Full - Copies all database files, filegroups, and transaction log data required to recover the database
- Contains the complete state of the database at the time the backup successfully finished
- the fastest and most efficient way to to RESTORE data
2) Differential - Captures onl the data (at the extent level) that has changed since the previous DIFF
backup
- Based on the latest FULL backup of a complete database, partial database, or set of
data files or filegroups

3) Log - Also called a Transaction Log Backup

- Captures all transaction log records that were not backed up in a previous log backup
- Note! The ability to perform transaction log backups depends on the database recovery model

4) Copy-only - Performs a backup without modifying backup settings that are normally modified
during conventional backups
- Does not effect differential or transaction log backup functionalities
5) File - Captures one or more database files or filegroups
6) Partial - Captures data from selected filegroups in a database

Understanding Recovery Models

- SQL Server backup and recovery operations occur within the context of the database recovery model
- What is a recovery model?
- A database property that controls how the transaction log is managed
- Determines how transactions are logged
- Determines what type of restore operations can be used
Three recovery models in SQL Server 2016
1) Simple - Transactions are written to the transaction log
- When the transaction is committed, database modification occurs
- The data is cleared from the transaction log
- The transaction log cannot be used for data recovery operations with this model
- Why use the simple recovery model?
- The data is not critical and can be easily recreated
- The database is only used for testing or development
- Losing any or all transactions since the last backup is not an issue
- What type of backups can be performed with the simple recovery model?
- Full backups
- Differential backups
- File or Filegroup backups
- Partial backups
- Copy-only backups

2) Full - SQL Server keeps all transaction data in the transaction log until a transaction log
backup occurs or the transaction log is truncated
- The transaction log contains all database modifications since the last transaction log
backup or truncation

- Why would you use the Full recovery model?

- Data is critical and cannot be lost
- Transaction log backups can be made on a frequent basis, capturing all modifications since the
last log backup
- Much faster backups
- Slower recovery time, since restoration requires the last full backup, then each
individual transaction log backup
- Point-in-time recovery can be performed

- Why would you choose the Full recovery model?

- The transaction log can grow to a very large size it it is not periodically backed up or truncated
- Can create disk space issues
3) Bulk-logged - Data that is loaded in bulk operations are logged with less detail
- The log only keeps track of the extents that have modified by the bulk load
operation
- BCP, BULK INSERT, etc… are bulk operations
- Transaction log sizes are smaller
- Note!
- Transaction log backups can be large when using bulk-logged recovery
- SQL Server back ups all data extents that have been modified by bulk load
operations since the last transaction log backup
- Can make point-in-time recovery impossible

Understanding Recovery Models

- How do you set the database recovery model?
- SSMSM GUI
- T-SQL ALTER DATABASE command
- ALTER DATABASE <db name> SET RECOVERY SIMPLE

Copy-Only Backups
- Backup plans for SQL Server usually involve a combination of backup types
- SQL Server determines which data to back up based on the type of backup being executed
- The backup makes a backup of all data
- A differential backup backs up only the data that has been modified since the last full backup
- How does SQL Server know which files have been updated?
- SQL Server stores data in extents
- An extent is a collection of eight 8 KB pages
- Each extent is mapped to a bitmap page
- By turning the bit on or off on the bitmap page, SQL Server can tell which extent has
been modified
- SQL uses the bitmap page to manage backups

- example : - Sunday
 - A FULL backup is performed on Sunday night
- All extents are backed up and the bitmap data is cleared
- Monday
- The bitmap bit is set to one for all extents that are modified during Mondays activity
- A Differential backup is performed on Monday night
- The Differential backup does not clear any bitmap bits

- If a failure occurs on Friday afternoon…

- Admins can restore the FULL backup from Sunday night and then restore the latest Differential
backup from Thursday night
- If there are no failures, another FULL backup takes place on Sunday night, and it resets all the bitmap
bits to zero, and the backup process takes place again through the week

Copy-Only Backups
- A Copy-Only backup performs a backup, but does not make any changes to the bitmap information
** this provides the ability to perform a backup without effecting the current backup plan in place **

Performed 3 ways:
1) SSMS
2) T-SQL
BACKUP DATABASE <db_name> TO DISK = <backup location.bak> WITH COPY_ONLY
3) Powershell
Backup-SQLDatabase -ServerInstance <server name>
-Database <db name> -BackupFile <path filename.bak>
-CopyOnly

Backup Terminology
- Backup device : a disk or tape device to which SQL Server backups are written to and from which they
can be restored
- SQL Server backups can also be written to a Windows Azure Blob storage service
( URL format is used to specify destination and file name )
- Backup media : one or more tapes or disk files to which one or more backups have been written
- Backup set : the backup content that is added to a media set by a successful backup operation
- Media family : backups created on a single non-mirrored device or a set of mirrored devices in a media
set
- Media set : an ordered collection of backup media, tapes or disk files, to which one or more backup
 operations have written using a fixed type and number of backup devices
- Mirrored media set : multiple copies (mirros) of a media set

System Database Backups

- SQL Server maintains a collection of system-level databases called system databases
- These are essential for the operation of a SQL Server instance
- Master ( should be backed up )
- Model ( should be backed up )
- Msdb ( should be backed up )
-Tempdb
- Distribution (if replication is being used) ( should be backed up )

- Limitations on restoring system databases

- System database backups are version specific
- Can be restored only from backups that are created on the same version of SQL Server
that the server instance is currently running
- the SQL Server instance must be running to restore a system database-level

- SQL Server will not start up if the master database is not accessible and usable
- the master database can be restored from a current backup, if the server instance will
start
- if the instance will not start, you can rebuild the master database

Transaction Log Basics

- Every SQL Server database has a transaction log
- it records all database modifications
- provides support and functionality for transactional operations
- multiple statements can be treated as a single unit
- all statements succeed or all fail

* - Data can exist in the transaction log in two forms

1) Committed transactions
- all commands in the transaction were successful and the modifications were applied to the
database
 2) Uncommitted transactions
- a transaction was started, but has not been rolled back or committed

- How the transaction log works

- All data modifications are written to the transaction log as a transactions
- the modifications can be ‘rolled back’, removed from the transaction log and not
applied to the database
OR
- the modifications can be committed, meaning they are applied to the database

- The transaction log stores a record of all recent activity

- Since the last time the transaction log was cleared
- Backed up ( backing up the log will clear it )
- Truncated
- Note! ( Full backups do not clear the transaction log )

- The transaction log data is also used when the SQL Server is restarted
- Planned or Unexpected

- When SQL Server starts up it consults the transaction log

- Some committed transactions may not have been written to the database due to the
failure/stoppage
- those transactions are applied to the database-l
- also called ‘rolled forwad’
- Uncommitted or incomplete transactions are ‘rolled back’ or removed from the
transaction log
- The process of consulting the transaction log during startup is called recovery
- We can take advantage of this process when backing up the restoring transaction logs

- Transaction Log Backups

- The transaction log contains a record of modifications to the database *****
- If the database recovery model is full or bulk-logged ****
- This makes the transaction log a very valuable asset for backup and recovery of database ***

*** NOTE : remember that having a database Recovery Model set to SIMPLE causes the transaction
log to automatically delete the log as data is moved into the database. ***
 - Once a FULL database backup has been completed, transaction log backups can occur
periodically.
- Provide similar functionality as an incremental backup
- Much smaller, much faster backup than FULL or Differential
- During a data restore, the full backup can be restored and backed up transaction logs
can be restored

- There are a few things to consider when restoring transaction log backups:
- The FULL backup and latest differential must be restored
- Each transaction log backup must be restored in the proper order using the WITH
NORECOVERY option
( this Prevents uncommitted transactions from being rolled back )
- then when restoring the Last transaction log we should include the WITH RECOVERY option
( this clears any uncommitted transactions and allows the SQL Server to recover to a
ready state )

- Transaction Logs can be backed up three ways:

1) SSMS
2) T-SQL
- Backup Log
3) PowerShell
- Backup-SQLDatabase cmdlet

Tail-Log Backups
- A Tail-Log backup is a transaction log backup
- A transaction log backup that captures any records that have not yet been backed up
- A “Tail -Log” backup is a transaction log backup that is performed if some kind of failure occurred and
you need that “data” that was in the transaction log that is after the most recent “Differential” backup
and before the next scheduled one.

- that way you could restore your backups like this:

- Full Backup → Differential Backup → Tail-Log Backup
** NOTE ! - If a Tail-Log backup cannot be performed, any transactions committed after the latest log
backup are lost.

Restoring Databases
- A database backup is useless if it cannot be restored !
- Restoring is the process of reading data from the backup and placing the data back into a useable state

Backup Plan
- A good backup plan includes step-by-step instructions on the restore process
- Location of backup files
- Location files should be restored to
- Etc.
- It is Vital to periodically perform a test restore from backups

- Restoring a backup can take different forms:

- Restore an entire database from a FULL backup
- Restore part of a database ( a partial restore )
- Specific files or filegroups ( a file restore )
- Specific pages ( a page restore )
- Restore a transaction log

- Restores can be performed in three ways:

1) SSMS
2) T-SQL
- Restore Database
- Restore Log
3) PowerShell
- Restore-SQLDatabase cmdlet

- Options to consider when performing a restore ( keywords )

- Recovery/Norecovery/Standby
- CheckSum/No_CheckSum
- Replace
- Partial
 - StopAt/StopAtMark/StopBeforeMark

- Recovery - Causes committed transactions in the log to be rolled forward and uncommitted
transactions to be rolled back
- Returns the database to a consistent, ready-to-use state
- NoRecovery – Does not roll back uncommitted transactions in the log
- The database remains unavailable for use
- this is used when we are in the process of restoring multiple logs and are not yet
finished or on the last log
- Standby – leaves the server in a read-only state
- used on a standby server when log shipping is being used
- CheckSum / NoCheckSum – Initiates the verification of both the backup checksums and page
checksums if checksums are present
- If no checksums are present, the restore proceeds without verification
- Replace - Replaces an existing database with data from a backup of a different database
- Without the REPLACE option, database checks are performed that prevents an existing
database from being overwritten
- Partial - Specifies a partial restore operation should be performed
- StopAt / StopAtMark / StopBeforeMark – Allow you to restore a database to a specific time or marked
location

Recovering The Master Database

- The master database serves as the ‘brain’ for a SQL Server instance
- It contains all the system tables that are necessary for the database server to functional
- The master database should be backed up periodically
- Also after significant modifications or updates

**
To restore the master database from a backup you must first start the server instance in a single-user
mode
- steps: 1) open SQL Server Configuration Manager
2) click on SQL Server Services
3) right-click on SQL Server in the right pane and choose properties
4) on the startup tab, enter -m in the startup parameters
5) restart the Database Engine
NOTE! - Stop the SQL Server Agent service to prevent it from starting up and using up your single
connection
**
- Can also use the sqlservr utility to start up in single-user mode
- browse to the location of the sqlservr.exe file in a command prompt window
- c:\program files\Microsoft SQL Server\MSSQL13.MSSQLSERVER\MSSQL\Binn
- execute: sqlservr.exe – m

Point-In-Time Recovery
- The transaction log stores a timestamp along with each data modification that is recorded
- We can use this information to perform a poin-in-time recovery
- Note! : the database must be using the FULL or BULK-LOGGED recovery models

- Why would we need to perform a point-in-time recovery?

- often, when an issue surfaces, the initial effects of the issue can be traced to a specific time
period
- users report that the database response time became slower at around 10am, for example

- How do we execute a point-in-time recover?

- use the STOPAT option in the RESTORE LOG T-SQL command
- RESTORE LOG <db name> FROM DISK = <backup file location> WITH FILE = <backup set
number>, NORECOVERY, STOPAT = ‘Apr 17, 2017 9:45 AM’

Log Shipping
- The transaction log can also be used to create a warm standby SQL Server
- A warm standby server is a server that does not provide instant, automatic switchover when a
failure occurs
- If the source server fails, the warm standby must be made available manually

- How log shipping works

- Transaction log backups are automatically sent from a primary database server to one or more
secondary servers
- The Secondary server restores the log using the WITH STANDBY option
- so.. essentially any update to the primary server causes the same update to be replicated on
each secondary server

- An optional 3rd type of server can be utilized to provide monitoring, history, and status information
 - it’s called a monitor server
- it raises alerts if the log shipping operation fails
- Log shipping offers another significant advantage
- the standby servers can provide read-only access to the data
- limited to the interval between restore jobs
- excellent for performing certain queries such as reporting

Performance and Monitoring

- Maintaining performance is an ongoing task due to constant, evolutionary changes

- more concurrent users
- more database dependent applications
- hardware/technology advancements
- business growth
- data growth

- Database performance is measured by two main criteria

1) Response Time
- the length of time between a user request for data and the completion of that request
- measured at the level of an individual transaction
2) Throughput
- the number of transactions that can be performed during a given time periodic

- Factors that affect database performance fall into 3 main categories

1) Applications
- applications that interact with the database require database resources
- inefficient T-SQL programming is a common culprit
- inefficient use of indexes or absence of indexes
- occur over time as new applications are introduced or existing applications are updated
2) System
- the physical database design can negatively affect performance
 - design requirements con evolve over time requiring more or less normalization
- the design and usage patterns of a database can increase or decrease resource locking
3) Resources
- hardware is always a major factor in performance
- CPU
- memory availability and performance
- disk I/O
- Network

Windows and SQL Server monitoring utilities and functionalities

- Performance Monitor
- graphical, real-time, monitoring of virtually every aspect of the Windows OS and SQL Server
- also provides the ability to capture performance metrics over extended time periods
- Dynamic Management Views ( DMVs)
- provides easy access to information relating to database performance
- provides current server state information
- these are Microsoft provided views in the “system databases”
- DBCC commands
- Database Consistency Calculator commands
- primarily used to check and monitor database consistency
- also used for maintenance tasks
- System stored procedures
- pre-programmed administrative and informational routines that can be called at will

Session Identification Overview

- the monitoring process will often identify performance issues
- once a performance issue is identified the source of the issue must be identified
- meet the SPID
- Every time an application connects to SQL Server, a new SPID is created
- Server Process ID
- Every SPID represents a session

- Session – is an occurrence of a user interacting with the database

- establishes a separate environment for the execution of user processes
- sessions provide the ability to allow multiple, simultaneous database connections and actions

Every SPID has these common characteristics

- An identification number
- A defined scope
- A defined memory space
- A connection string
- Cannot interact with other SPIDs

- the System Stored Procedure sp_who can be used to get information about sessions in SQL Server
- example:
run this SQL to get your own SPID: select @@spid

run this SQL to get a list of sessions to find your SPID: sp_who

run this SQL to see only sessions that belong to specific SPID: sp_who ‘SPID’
sp_who ‘52’
run this to see all sessions related to a single login name: sp_who ‘ login name here’
sp_who ‘WIN-043S00EVCVN\Mat Wang’

Identifying Blocking Sessions

- at any one time, the SQL Server database engine must manage the actions of multiple simultaneous
users
- each user must be able to confidently select or modify data
- SQL Server accomplishes this through a process called locking
Locking – is managed at the transaction level, which is identified and managed at the session
level
How Locking Works
- user A connects to SQL Server and begins a transaction that will update a row in the Parts table
- SQL Server then puts a lock on that row
then…
- user B attempts to modify the same row
- the lock on the row prevents user B from accessing the row
 - user B experiences a wait condition until user A’s transaction is completed
- locking is a necessary function in most database operations !!
Note! - in-memory tables do not experience locking

When locking becomes excessive, database performance suffers

- users experience blocks = they must wait for access to data due to locking

Excessive locking can be caused by a number of situations:

- not enough hardware resources
- inefficient physical database design
- inefficient T-SQL programming
- improperly managed transactions
- long running queries
- etc..

Blocking sessions can be identified using two tools

1) Activity Monitor in SSMS
- can use this to identify the process blocking another, but you can also use it to kill a process
2) sp_who system stored procedure ( also newer sp_who2 )

Understanding UCP’s
- Enterprise level organizations utilize multiple SQL Server instances
- In a multi-server environment, the administrator needs to be able to efficiently monitor the overall
health and utilization levels of each SQL Server

- SQL Server 2008 R2 introduced the SQL Server Utility and Utility Control Point (UCP)
- models an organization’s SQL Server-related entities in a unified dashboard-style view

- A Utility Control Point (UCP) is part of the SQL Server Utility tool
- SQL Server Utility is managed through a UCP using the Utility Explorer in SSMS
- The UCP provides the central reasoning point for the organization and monitoring of SQL Server
health
- What does a UCP do?
- collects configuration and performance information from managed instances of SQL Server every 15
minutes
- the collected data is stored in the utility management data warehouse ( UMDW ) on the UCP
- the data warehouse is named sysutility_mdw
- the UCP supports actions like specifying resource utilization policies
- track the utilization requirements of your organization
- help identify resource use bottlenecks and consolidation opportunities

** note ** SQL Server Utility requirements:

- SQL Server Enterprise edition ( can be tested on SQL Server Developer edition )
- the SQL Server instance type must be Database Engine
- the SQL Server Utility must operate with a single Windows domain, or across domains with
two-way trust relationships
- the SQL Server Utility service accounts on the UCP an all managed instances of SQL Server must
have read permission to Users in Active Directory
- the UMDW database on the UCP will consume approx… 2 GB per managed instance per year

Data Collector Basics

- SQL Server Data Collector is a feature for performance monitoring and tuning
- accessible in SQL Server Management Studio ( SSMS )
- introduced in SQL Server 2008
- available in the Enterprise, Standard, and Web editions
- Data Collector provides one central point for data collection across your database servers and
applications
- collects performance information from multiple SQL Server instances and stores it in a single
repository, the Management Data Warehouse ( MDW )

- it collects three built-in data collection specifications

- collects basic, important performance metrics by default:
- Disk usage
- Query statistics
- Server activity

- Data Collector works by creating and running scheduled jobs (needs SQL Server Agent running )
- also stores collected data for alter analysis with built-in reporting
- utilizes SQL Server Integration Services packages to transform and load collected data into
Management Data Warehouse ( MDW )
Data Collector Configuration
Four main steps:
1) Configure the Management Data Warehouse
2) Configure the properties of a data collector
3) Enable (or disable) data collection
4) View a collection set report

Query Store Basics

- the SQL Server Query Store is a new feature in SQL Server 2016
- it is one of the most significant enhancements in 2016
What is the Query Store?
- a utility designed to greatly simplify the task of troubleshooting query performance
- it Captures query execution information
- number of executions
- average duration
- etc..
- captures every execution plan that has been generated for a specific query
( the DBA can decide which execution plan that SQL Server should use for that specific query )

Common uses for the Query Store

- quickly find an fix a plan performance regression by forcing the previous query plan
- determine the number of times a query was executed in a given time window
( excellent when troubleshooting performance resource problems )
- identify top <n> queries in the past <x> hours
( by execution time, memory consumption, etc.. )
- audit the history of query plans for a given query
- analyze the resource (CPU, I/O, and memory) usage patterns for a particular database

** the Query Store is disabled by default **

- must be enabled:
- Right-click a database → Properties → Query Store
 or
- ALTER DATABASE <db name> SET QUERY_STORE=ON
Understanding Extended Events
- SQL Server Extended Events provide the ability to monitor SQL Server based on specific events
- Data is captured when a specific event occurs
( instead of capturing a large amount of data and searching through it )
- Performs data collection in a lightweight manner
( thus it has less impact on SQL Server performance )

- Extended events provides the ability to see details about specific inner operations
- When creating an extended event session, you specify two main metrics:
1) which particular occurrences you are interested in
2) how you want the data reported to you

Extended events can be configured three ways:

1) SSMS
2) T-SQL
3) Dynamic Management Views ( DMVs ) that target extended events
- the SSMS interface for Extended Events is excellent

** The ability to create Extended Events requires a user account that has the server permission of ALTER
ANY EVENT SESSION **

Creating an Extended Event

- done in SSMS under ( object explorer ) → Management → Extended Events