PowerBI G2-v3

SQL BI
Co-op Project
Group 2
Members:
Chuyi Zhang (Group Lead)
Annie Song
Felix Chen
Project Overview
This project is designed to perform the upgrade of some major
technologies used to support the course for Jay Liu at Victoria
Training Center, and this document describes the detailed work items
required for this technology upgrade.
This requirement document is to request working on the following
technology enhancement or migration, to modernize, optimize, and
consolidate the existing version of the course, Microsoft Power BI
(Power BI). With the project completion, the enhanced features
running on the upgraded system design will offer a better designed,
documented, improved, and standardized environment, which
supports a new look of the technologies the course runs on, and offer
a nicely-looking infrastructure of the new version of the course.
2
Task Distribution
No. Contents Performer
Virtual Machine (C1, C4B)
2.1 & 2.2 Skip
Using VM as a Server (C1 – C7)
Enlarging Dataset to 20 Million and Up-to-Date (C1
2.3 & 2.4 – C3) Chuyi Zhang
Dynamic Increase of Up-to-Date Data (C3 – C5)
2.5 More Data Sources (C1 – C4A) (Optional) Chuyi Zhang
Python in Power BI (New, maybe C5, C6)
2.6 & 2.7 Felix Chen
R in Power BI (New, maybe C5, C6)
Adding More KPI with Charts in Reports (C4, C5) Annie Song,
2.8 & 2.9
Adding More Geometrical Reports (C4, C5) Chuyi Zhang
Report Deployment (C4, C5)
2.10 & 2.11
Version Consolidation (C1 – C7)
2.12 Rebuild All Technical Documents (PPT) All members
3
2.1 -2.2 Virtual Machine (C1, C4B)
& Using VM as a Server (C1 – C7)
Target
 Use only one Virtual Machine for the whole process
 Use VM as a purely server to support Power BI reports
development all on the desktop
4
2.3 Enlarging Dataset to 20 Million and Up-to-
Date
(by Chuyi Zhang)
Target
 Using the VM as a server/servicing
 Increasing data/database size from the 5M rows in total to
up to 20 million
 Let data up-to-date to build reports in Power BI
5
Date
(by Chuyi Zhang)
 Work on table [Orders]
• Create table [Receivers] with columns [ShipName], [ShipAddress],
[ShipCity], [ShipRegion], [ShipPostalCode], [ShipCountry],
[CreditCard] from table [Orders], and its own identity numeric
column [ReceiverID].
• Create table [CustomerRownum] with column [CustomerID] from
table [Customers] and its own identity numeric column
[CustomerRN]
Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Check details in file: PowerBI 2.3 Orders -Chuyi Z.sql
PowerBI 2.3 OrderDetails -Chuyi Z.sql 6
Date
(by Chuyi Zhang)
• Create table [OrdersSource01], keep column [OrderID] from table
[Orders], get random value for numeric columns [CustomerID],
[EmployeeID], [Shipvia], [ReceiverID]
• By joining tables:
- Replace numeric column [CustomerID] by text column
[CustomerID]
- Replace numeric column [ReceiverID] by columns [ShipName],
[ShipAddress], [ShipCity], [ShipRegion], [ShipPostalCode],
[ShipCountry], [CreditCard]
- Result to table [OrderSourceCR]
Date
(by Chuyi Zhang)
• Create table [OrderDateEn] to get value for columns [OrderDate],
[RequiredDate], [ShippedDate]:
- Suppose when OrderID increase 1, OrderDate increase 0 or 1
randomly, set the dates for the recent 20 years (2000 to 2019);
- Suppose suppose RequiredDate is 0 to 3 days later than
OrderDate randomly;
- Suppose ShippedDate is 0 to 7 days later than RequiredDate
randomly
- Get OrderID by identity column

Date
(by Chuyi Zhang)
• Join tables [OrderDateEn] and [OrderSourceCR] to get the final table
[OrdersEnlarge] with1,444,700 rows, add primary key and clusted
index.
Finish the enlargement and aging for table [Orders]

Date
(by Chuyi Zhang)
 Work on table [OrderDetails]
• Create table [OrderDetails001], enlarge columns [OrderID] and
[ProductID] with random value
• Union tables [OrderEnlarge] and [OrderDetails001] on columns
[OrderID] and [ProductID] to ensure no OrderID in [OrderEnlarge]
was missed

Date
(by Chuyi Zhang)
 Work on table [OrderDetails]
• - Remove duplicate of the combination of [OrderID] and [ProductID];
- Get UnitPrice value from table [Products];
- Get random value for [Quantity] and [Discount], the value range is
from table [OrderDetails]
- Add combine Primary Key [OrderID] and [ProductID]
- Result to final table [OrderDetailEn] with 11,444,115 rows
Finish the enlargement and aging for table [OrderDetailEn]

2.4 Dynamic Increase of Up-to-Date Data
(by Chuyi Zhang)
Have a linked server with the

‘Microsoft.ACE.OLEDB.12.0’
OLE DB provider
• Download and install
‘AccessDatabaseEngine’
• Right click on the ’Linked
Servers’ folder and choose
the ’New Linked Server’
12
(by Chuyi Zhang)
• Fill ’New Linked
Server’ dialog as
showed
- Linked Server:
- Provider:
- Product name
- Data source
- Provider string
• Click OK
13
(by Chuyi Zhang)
• In created linked server
‘EXCELDATA’, the Excel data
source file’ Sheet(s) show
under ‘Tables’
• Create the linked server by
execute stored procedure
• Be sure the configuration of the
‘SQL Server Properties’ – ‘Log
On’ is ‘Local System’

Check details in file: PowerBI 2.4 -Chuyi Z.sql
14
(by Chuyi Zhang)
 Import Up-to-Date Data to database
• Suppose the enlarged and aged tables [OrdersEnlarge],
[OrderDetailEn] are up-to-date, only need to upgrade
daily
• Create up-to-date tables from them
• Add constraint to up-to-date tables
• Import current data from Excel source file by stored
procedure [dbo.usp_Uptodate_Orders_Details]
15
(by Chuyi Zhang)
 Dynamic increase
data by job scheduled
• Create a new job
Object Explorer
-> SQL Server Agent
-> Jobs
-> Right click to create a
new job
-> Job Properties
-> In ‘General’, name it as
‘UpgradeDaily’
16
(by Chuyi Zhang)
• Add new step(s)
17
(by Chuyi Zhang)
• Add new step(s)
- Create a Step name
as ‘UptoDate’
- Type: T-SQL
- Database: Orders
- Command:
Execute
usp_Uptodate_Orders_
Details
18
(by Chuyi Zhang)
 Add new
Schedules
19
(by Chuyi Zhang)
 Add new schedules
- Name
- Schedule type
- Frequency
- Duration
- Summary
20
(by Chuyi Zhang)
 (Optional) Check and show new upgraded data (current
date), specific date’s data
• check and show current date upgrade data:
[dbo.usp_UpgradeShow]
• check and show special date ('yyyy-mm-dd') upgrade data
[dbo.usp_CheckShow]

21
2.5. (Optional) More Data Sources (C1 –
C4A) (by Chuyi Zhang)
Target:
Try to involve onling cloud-based sources into the project,

such Azure Consumption Insights, Google Analytics,
Facebook, GitHub
22
• In SSMS, create view [vw_USProductionSourceOrder].
There are 2-letter code as abbreviations for U.S. states
name in [Region] column.
• In Power BI, Import Mode, import view
[vw_USProductionSourceOrder]
• Get data in ‘Web’ data source, enter the link
https://en.wikipedia.org/wiki/List_of_U.S._state_abbreviati
ons
Check code in file: Power BI 2.5(optional)-Chuyi Z.docx
Check details in file: PowerBI 2.5(optional) -Chuyi Z.sql
PowerBI 2.5(optional)-Chuyi Zhang.pbix 23
• choose tabel ‘Codes and abbreviations for US states,
federal district, territories, and other’ to load
24
• Rename columns that you need to
• On column ‘USPS’, set Data category to ‘State or
Province’
25
• Go to ‘Relationships’, create relationship between
columns [ProductRegion] and [USPS]
26
Create ‘Matrix’ and ‘Map’
27
2.6 Python in Power BI (New, maybe C5, C6)
• Install python and package

• Extract data from SQL server
• Import data in Power BI using Python
• Visualize data in Power BI using Python
28
Install python and package
• To run Python scripts in Power BI Desktop, you need to install Python on
your local machine. You can download Python from the Python website.
The Power BI Python integration requires the installation of two Python
packages: pandas, matplotlib.
29
Extract data from SQL server
• To prep the data from SQL server, here I exported the orders table from the
orders database as txt file with 2,241,124 rows called Orders_orders.txt
30
Import data in Power BI
using Python
• Import the txt file in Power BI using

Get Data, choose Python Script and
input:data =
pandas.read_csv('C:\Orders_Orders.tx
t')
31
32
Visualize data in
Power BI using
Python
• Our goal here is to plot a chart
showing the number of orders by
year.
• Select Python visual from the pane
and drag Year, OrderID into Values,
then change OrderID to count.
33
• A script editor will pop up automatically, then key in the codes below, and click the run
button.
import matplotlib.pyplot as plt
plt.plot(dataset.Year,dataset.OrderID)
plt.xlabel('Year')
plt.ylabel('Count of Orders')
plt.title('Count of Orders by Year')
plt.show() 34
• Right figure shows the
number of orders by
year chart.
35
2.7. R in Power BI (New, maybe C5, C6)
• Install R and package
• Extract data from SQL server
• Import data in Power BI using R
• Visualize data in Power BI using R
36
Install R and package
• To run R scripts in Power BI Desktop, you need to install R on your local
machine. The Power BI R integration requires the installation of R package:
ggplot2
37
Extract data from SQL server
• To prep the data from SQL server, here I exported the orders table from the
orders database as txt file with 2,241,124 rows called Orders_orders.txt
38
Import data in
Power BI using R
• Import the txt file in Power
BI using Get Data, choose R
Script and input:
• Data <-
read.csv("C:\\Orders_Order
s.txt", sep=",")
39
40
Visualize data in
Power BI using R
• Our goal here is to plot a chart
showing the number of orders by
year.
• Select R visual from the pane and
drag Year, OrderID into Values, then
change OrderID to count.
41
• A script editor will pop up automatically, then key in
the codes below, and click the run button.
library(ggplot2)
ggplot(data=dataset, aes(x=Year, y=OrderID)) +
geom_line() +
ggtitle("Count of Orders by Year") +
labs(x="Year", y="Count of Orders")
42
Below figure shows the number of orders by year
chart.
43
2.8: Adding More KPI with Charts in Reports
Task requirements:
Add three more reports to show KPI with proper visualization by
using both DAX-based and database-based solutions.
as below:
• Employee Average Sales (Amount) Performance with

Visualization 1
• Customer Average Shopping (Spending) Performance with

Visualization 2
• Shipping Company Average Delivery (Duration) Performance

with Visualization 3
44
2.8.1: Employee Average Sales Amount
Create a formula to obtain the employee average sales amount by

using compute the employee average sales amount at highly
aggregated level. The calculation is below:
The average value of current year 2019 is compared with last year
2018, in the sense of percentage change to evaluate the
performance of employee average sales amount is calculated by
(Ave_2019 – Ave_2018) / Ave_2018.
45
Steps:
Go SSMS, execute T-SQL scripts to generate an aggregated table, then
loads the table into PBI for creating the KPI.
Script:
Create a view “vw_KPI_Employee_Average_Sales”.
46
Go Power BI Desktop (PBI), click “Get data”, select SQL Server database,
type “VM-DA-BI\POWERBI” as server name, and “Orders” as database
Name, check “Import” as data connectivity mode, “OK”.
Go“Navigator”, check “vw_KPI_Employee_Average_Sales”, then click
“Load”, the table is loaded in PBI.
Go “Modeling” tab at ribbon, click on “New Column”, type in “Target =
0.3”, which means that increase in employee average sales amount.
Go report canvas at right side, select “KPI” in “Visualization” tab.
Set the parameter values.
47
Now use Dax:
Go PBI, get data from SQL Server database, which selected [Orders] and
[Order Details] to load.
48
Build two measures for employee average sales amount for current year
2019 and last year 2018.
49
Create two more measures, the current year number and current year
percentage change on employee average sales amount, these will be used
to build KPI.
The results:
Two KPI, created by DAX and by T-SQL, are same.
50
2.8.2: Customer Average Shopping
Spending
Create formula first, compute the customer average spending
amount and the average value of current year 2019 is
compared with 2018, the performance on customer average
spending amount is calculated by (Ave_2019 – Ave_2018) /
Ave_2018.
Calculate year of 2019 first, then year of 2018, load two tables
to PBI.
51
2.8.2: Customer Average Shopping Spending
T-SQL script:
52
Spending
53
Use DAX, load two tables [Orders] and [Order Details] to
PBI.
Go “Table” tab, click on [Order Details], select “New Column”

at “Modeling” , create formula to generate column “Amount”.
54
Spending
Create two more measures, the current year of 2019 number and
current year percentage change on customer average purchase
amount.
There are NULL values in CustomerID, move those orders to an

existing Customer. The results are same by SQL and DAX.
55
2.8.3: Shipping Company Average Delivery Duration
Use “ShippedDate” as “Delivery date” , then apply the function

“DATEDIFF” (shippeddate – orderdate), average of date difference over
number of orders, ROUND(), and CAST() function; create view, then load
it into PBI Desktop.
Use AVG() function to calculate the average of dates for current year 2019
and last year 2018.
Purpose: The company tries to shorten delivery duration that the average
delivery duration.
56
57
Go PBI, use DAX, load [Orders] and [Order Details] tables from SQL
server, select [Orders] table to create a new column called “Duration”.
58
2.9: Adding More Geometrical Reports
Task requirements:
Add three more reports to show performance with geometrical

reports:
• Employee-location based sales with Geometric Visualization 1
• Customer-location based spending with Geometric

Visualization 2
• Product-location based ordered with Geometric Visualization

3
For those reports developed, we would like to have both DAX-

based and database-based solutions.
59
2.9.1: Employee-location based sales
Calculate the employee average sales amount of grouped by

countries based on year 2019 data through using SQL
database-based solution and DAX base.
60
Go PBI, load the view to PBI Desktop:
61
Go report canvas, drag “ShipCountry” to the blank area and drop it
at the field we designed to, creates a map with “bubbles”.
Drag “Employee Avg Sales” to “Size”, select the calculation type

to “Average”.
Use same source to create column chart and table report.
Go report canvas, click on a specific country on the map, table, or

column chart, then that country’s information will be spotlighted
automatically. Also can select specified countries.
62
63
64
Now use DAX, load [Orders] and [Order Details] from SQL
Server to PBI, add two measures by using DAXs.
Go report canvas to drag “ShipCountry” to the blank area,

and drag “Current_Year_Average” to “Size”, then create the
table and column chart.
Generated by DAX is exactly the same as generated by T-

SQL.
65
2.9.2: Customer-location based spending
Calculate the customer average purchase amount of grouped
by countries for year 2019 by using two methods: database-
based method and DAX.
Create a view by script below, execute this view and load to

PBI.
66
Create the report is as follows:
67
Use DAX, load [Orders] and [Order Details] from SQL Server to
PBI, create two measures as follows:
Drag “ShipCountry” to the blank area in report canvas, then drag

“Current_Year_Average” to “Size”, then create the table and
column chart.
68
The report is exactly the same by using two methods SQL and
DAX.
69
2.9.3: Product-location based ordered
Calculate the average number of product per order of grouped by
countries for year 2019. Use two methods.
First, database-based method, credit a view by script as follow:
70
2.9.3: Product-location based ordered
Report generated by T-SQL below is same like using DAX.
71
2.8. Adding More KPI with Charts in Reports
(Complement) by Chuyi Zhang
 DAX-based solutions:
 In SSMS, create view [vm_KPIsource01] as
data source, to reduce the data size that need to
import into Power BI and to avoid touching up-to-
date data source directly
 Use Import Mode in Power BI, import view
[vm_KPIsource01] as data source
 Create reports in Power BI
Check code in file: Power BI 2.8-2.9 v3 - Chuyi Z.docx

Check details in file: Sql PowerBI 2.8-2.9 v3.sql
PowerBI 2.8-2.9 DAX-Chuyi Zhang.pbix
PowerBI 2.8-2.9 DirectQuery-Chuyi Zhang.pbix
72
2.8. Adding More KPI with Charts in Reports
 Database-based solutions:
 In SSMS, create views as data source, to
reduce the data size that need to run in Power BI
and to avoid touching up-to-date data source
directly
 Use DirectQuery Mode in Power BI

73
2.8.1: Employee Average Sales Amount (Import Mode)
Create Table [KPI_Employee Average Sales] using DAX query:
Use Summarizecolumns function, list Year, Employee amount
per year, and total Sales per year.
Sort ascending
by ‘Sales Year’
74
Add column ‘Employee Average Sales’ by ‘Yearly Sales’ / ‘Employee

Amount’
Sort ascending
by ‘Sales Year’
75
Create ‘column chart’ and ‘table’ from [KPI_Employee Average Sales]
76
2.8.1: Employee Average Sales Amount (DirectQuery Mode)
Create ‘column chart’ and ‘table’ from view [vw_KPIemployeeAveSales]
77
2.8.2: Customer Average Shopping Spending (Import Mode)
Create Table [KPI_Customer Average Spending] using DAX query.
Use Summarizecolumns function, list Year, Customer amount per
year, and total Sales per year.
Sort ascending
by ‘Sales Year’
78
Add column ‘Customer Average Spending’ by ‘Yearly Sales’ / ‘Customers
Amount’
Sort ascending
by ‘Sales Year’
79
Create ‘column chart’ and ‘table’ from [KPI_Customers Average Spending]
80
(DirectQuery Mode)
Create ‘column chart’ and ‘table’ from view
[vw_KPIcustomerAveShopping]
81
(Import Mode)
Create Table [KPI_Shipping Average Delivary] using DAX query.
Use Groupby function, list Year, average ShipDuration
Sort ascending
by ‘Year’
82
(Import Mode)
Create ‘column chart’ and ‘table’ from [KPI_Shipping Average Delivary]
83
(DirectQuery Mode)
Create ‘column chart’ and ‘table’ from view
[vw_KPIcompanyAveDelivery
84
2.9. Adding More Geometrical Reports
 DAX-based solutions:
 In SSMS, create views [vm_KPIsource01] and
[vw_ProductionLocationSourceOrder] as data
source, to reduce the data size that need to import
into Power BI and to avoid touching up-to-date data
source directly
 Use Import Mode in Power BI, import views
[vm_KPIsource01] and
[vw_ProductionLocationSourceOrder] as data source
85
2.9. Adding More Geometrical Reports
 Database-based solutions:
 In SSMS, create views as data source, to
reduce the data size that need to run in Power BI
and to avoid touching up-to-date data source
directly
 Use DirectQuery Mode in Power BI
86
2.9.1. Employee-location based sales (Import Mode)
Create ‘Matrix’ and ‘Map’ from views [vm_KPIsource01]
87
2.9.1. Employee-location based sales (DirectQuery Mode)
Create ‘Matrix’ and ‘Map’ from views [vw_EmployeeLocationSales]
88
2.9.2. Customer-location based spending (Import Mode)
Create ‘Matrix’ and ‘Map’ from views [vm_KPIsource01]
89
2.9.2. Customer-location based spending (DirectQuery Mode)
Create ‘Matrix’ and ‘Map’ from views [vw_CustomerLocationSpending]
90
2.9.3. Product-location based ordered (Import Mode)
Create ‘Matrix’ and ‘Map’ from views vw_ProductionLocationSourceOrder]
91
2.9.3. Product-location based ordered (DirectQuery Mode)
Create ‘Matrix’ and ‘Map’ from views [vw_ProductLocationOrder]
92
2.10 Report Deployment (C4, C5)
(by Chuyi Zhang)
Target:
 Deploy reports developed in 2.8 and 2.9 on cloud (Power
Service on powerbi.microsoft.com)
 Deploy reports developed in 2.8 and 2.9 on premises (Power

Report Server in local)
Refresh should be scheduled in order to updated automatically

and periodically.
93
(by Chuyi Zhang)
 Deploy reports on premises
In ‘DirectQuery’ Mode, File -> Options and Settings -> Options ->
Preview features -> Select ‘Automatice Page Refresh’
94
(by Chuyi Zhang)
 Deploy reports on premises
At the report page that need to enable refresh,
‘Visualizations’ pane -> ‘Formatting’ -> ‘Page
refresh’ (near the bottom of the pane)
- Turn ‘Page refresh’ on
- Select Refresh type ‘Auto page refresh’
- Set Duration
95
(by Chuyi Zhang)
 Deploy reports on cloud
• In ‘DirectQuery’ Mode,
Home -> Pulish
• Sign in
• Repeat the same steps
as in Power BI Desktop
‘Visualizations’ pane
96
2.11 Version Consolidation (C1 –
C7)
Target: (by Chuyi Zhang)
Catch up the newest version of Power BI Desktop across
the entire project
Steps:
 Download the newest version of Power BI Desktop
 Reinstall Power BI Desktop
 Open it, check options and settings, such as refresh
settings, to match what you need
 Update other software applications if need, such as SSMS
97

PowerBI G2-v3

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PowerBI G2-v3

Uploaded by

Copyright:

Available Formats

SQL BI

2.12 Rebuild All Technical Documents (PPT) All members

 Use VM as a purely server to support Power BI reports

development all on the desktop

 Increasing data/database size from the 5M rows in total to

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Have a linked server with the

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

• Add new step(s)

Check code in file: Power BI 2.3-2.4 -Chuyi Z.docx

Try to involve onling cloud-based sources into the project,

• Install python and package

• Import the txt file in Power BI using

• Employee Average Sales (Amount) Performance with

• Customer Average Shopping (Spending) Performance with

• Shipping Company Average Delivery (Duration) Performance

Create a formula to obtain the employee average sales amount by

Go “Table” tab, click on [Order Details], select “New Column”

There are NULL values in CustomerID, move those orders to an

Use “ShippedDate” as “Delivery date” , then apply the function

Add three more reports to show performance with geometrical

• Employee-location based sales with Geometric Visualization 1

• Customer-location based spending with Geometric

• Product-location based ordered with Geometric Visualization

For those reports developed, we would like to have both DAX-

Calculate the employee average sales amount of grouped by

Drag “Employee Avg Sales” to “Size”, select the calculation type

Use same source to create column chart and table report.

Go report canvas, click on a specific country on the map, table, or

Go report canvas to drag “ShipCountry” to the blank area,

Generated by DAX is exactly the same as generated by T-

Create a view by script below, execute this view and load to

Drag “ShipCountry” to the blank area in report canvas, then drag

First, database-based method, credit a view by script as follow:

Check code in file: Power BI 2.8-2.9 v3 - Chuyi Z.docx

Check code in file: Power BI 2.8-2.9 v3 - Chuyi Z.docx

Add column ‘Employee Average Sales’ by ‘Yearly Sales’ / ‘Employee

Create ‘column chart’ and ‘table’ from [KPI_Employee Average Sales]

Create ‘column chart’ and ‘table’ from view [vw_KPIemployeeAveSales]

Create ‘Matrix’ and ‘Map’ from views [vm_KPIsource01]

Create ‘Matrix’ and ‘Map’ from views [vm_KPIsource01]

Create ‘Matrix’ and ‘Map’ from views [vw_CustomerLocationSpending]

Create ‘Matrix’ and ‘Map’ from views vw_ProductionLocationSourceOrder]

Create ‘Matrix’ and ‘Map’ from views [vw_ProductLocationOrder]

 Deploy reports developed in 2.8 and 2.9 on premises (Power

Refresh should be scheduled in order to updated automatically

- Turn ‘Page refresh’ on

- Select Refresh type ‘Auto page refresh’

You might also like