Power Query - Reference Book PDF

Master Power Query Power Query – Reference Guide
Table of Contents
Part 1
Power Query Basics
Power Query Editor ........................................................................................................................ 3
Overview ................................................................................................................................................ 3
Updating the Source Location/Name .................................................................................................... 4
Deleting a Query Step ............................................................................................................................ 5
Transform Versus Add Column .............................................................................................................. 6
Split Text by Delimiter ........................................................................................................................... 7
Automatic Data Typing .......................................................................................................................... 7
Rename a Column.................................................................................................................................. 8
Removing (Deleting) a Column .............................................................................................................. 8
Perform Quick Mathematical Operations ............................................................................................. 9
Refreshing Source Data During Query Composition ............................................................................. 9
Quick Insights on Data – Column Profiling, Data Quality & Distribution ......................................... 10
Displaying a Complete List When Filtering .......................................................................................... 10
Displaying Column Profile Information ............................................................................................... 11
Displaying a Complete List for Selected Column ................................................................................. 12
Column Distribution............................................................................................................................. 12
Column Quality .................................................................................................................................... 13
Using Monospaced Fonts .................................................................................................................... 13
Discover the Total Number of Rows .................................................................................................... 14
Formula Bar – Applied Steps & M Code ......................................................................................... 15
Renaming Applied Steps ...................................................................................................................... 15
Other Applied Steps Options ............................................................................................................... 15
Adding Documentation to Applied Steps ............................................................................................ 16
Power Query Formula Bar ................................................................................................................... 16
Advanced Editor................................................................................................................................... 17
Close & Load Destinations ............................................................................................................. 18
Working with Existing Queries............................................................................................................. 18
Creating a Data Connection ................................................................................................................. 19
Changing the Default Load Destination ............................................................................................... 19
“Saving” Your Queries ......................................................................................................................... 20
Copyright © 2020 Leila Gharani

Refresh Data and Refresh Options................................................................................................. 21

Refresh Options ................................................................................................................................... 21
Part 2
Important Power Query Tips & Tricks
Uploading Data from Excel ........................................................................................................... 25
Uploading from Tables ........................................................................................................................ 25
The Query Ribbon ................................................................................................................................ 26
Useful Table Settings ........................................................................................................................... 26
Uploading from Named Ranges ........................................................................................................... 26
Handling Changes to Source.......................................................................................................... 27
Correcting Changes in Location or Filename ....................................................................................... 27
Correcting Changes to Sheet Names ................................................................................................... 28
Correcting Changes to Table Names ................................................................................................... 29
Correcting Changes to Column Heading Names ................................................................................. 30
Correcting Changes Using the Advanced Editor .................................................................................. 31
Understanding Data Types ............................................................................................................ 32
The Importance of Data Types ............................................................................................................ 32
Data Types vs. Formatting (and NULL Formatting) ........................................................................ 33
Power Query Examples ........................................................................................................................ 33
Invoking Automatic Data Types ........................................................................................................... 35
Working with NULLs ............................................................................................................................ 35
Power Query Shortcuts ................................................................................................................. 36
Common time-saving features ............................................................................................................ 36
Error Handling – Finding & Correcting Errors in Data ..................................................................... 38
Errors that should be corrected at the source level ............................................................................ 38
Determining which rows contain errors .............................................................................................. 38
Errors to correct in Power Query ........................................................................................................ 39
Methods to Deal With Errors in Power Query .................................................................................... 40
More Data Views – Duplicate or Reference Query?........................................................................ 41
How to Duplicate a Query ................................................................................................................... 41
How to Reference a Query .................................................................................................................. 42
Query Dependency – Visualize Query Relations ............................................................................. 44
Viewing Query Dependencies Graphically .......................................................................................... 44
Query Management – Delete, Manage, Copy Queries & Backup Results ........................................ 45
Delete a Query ..................................................................................................................................... 45

Copy a Query to Another Workbook ................................................................................................... 47
Export a Query for Later Use ............................................................................................................... 47
Organizing Queries .............................................................................................................................. 48
Part 3
Helpful Power Query Transformations
Text Transformations – Format, Extract & More ............................................................................ 51
Transformation #1: Split Text by Delimiter ......................................................................................... 51
Transformation #2: Extract Employee ID from Unique Code .............................................................. 54
Transformation #3: Separate Names and Format Casing ................................................................... 57
Merge Columns – What to Watch Out For ..................................................................................... 63
Selecting the Columns to Be Merged .................................................................................................. 63
Merging Without Extraneous Spaces .................................................................................................. 64
Capturing the First Letter Only In the Middle Name ........................................................................... 65
Standardizing Middle Initial Casing & Style ......................................................................................... 65
Fill & Replace Values – To Create a Proper Dataset ....................................................................... 66
Repeat Cells to Adjacent Cells (Fill) ..................................................................................................... 66
Replacing Data ..................................................................................................................................... 67
Standardizing Casing ............................................................................................................................ 68
Replacing NULL Values......................................................................................................................... 68
Sort Data – Including Multiple Levels ............................................................................................ 69
Simple Sorts ......................................................................................................................................... 69
Multi-Level Sorts .................................................................................................................................. 69
Returning to an Original Sort Order .................................................................................................... 70
Remove Duplicates – Including Multiple Columns .......................................................................... 71
Remove Duplicates Based on a Single Column.................................................................................... 71
Remove Duplicates Based on Multiple Columns ................................................................................. 71
Number Transformations – What to Watch Out For ...................................................................... 72
Adding a Set Value to All Values in a Column ...................................................................................... 72
Rounding Values in a Column (with little know feature) ..................................................................... 72
Produce an Aggregation Using Two Columns ..................................................................................... 73
Adding Columns That Contain Nulls .................................................................................................... 73
Multiply a Percentage Against a Value ................................................................................................ 74
Preserving the Top N/Bottom N Rows Based on a Column ................................................................ 74
Working with Filter – AND & OR Conditions .................................................................................. 75

Filtering by Specific Items .................................................................................................................... 75
Filtering by Text That Exists ................................................................................................................. 75
Creating a Compound Field Filter ........................................................................................................ 76
Beware of Logic Errors ......................................................................................................................... 77
Filter Selection Trap ...................................................................................................................... 78
Predicting the Future ........................................................................................................................... 78
Change Type Trap – (& Remove Columns Trap) ............................................................................. 80
Change Type Trap – Problem............................................................................................................... 80
Change Type Trap – Solution ............................................................................................................... 81
Adjusting Automatic Change Type in Power Query Settings .............................................................. 81
Remove Columns Trap – Problem ....................................................................................................... 82
Remove Columns Trap – Solution ....................................................................................................... 82
Part 4
Powerful Power Query Transformations
Column From Example – to Extract Patterns Quickly ...................................................................... 85
Formulas For Free ................................................................................................................................ 87
Solving the Problem with the Traditional Approach ........................................................................... 87
Conditional Columns – in Power Query .......................................................................................... 89
If… Then… Else Statements in Power Query ....................................................................................... 89
Aggregating (Grouping) Data – on Multiple Levels ........................................................................ 91
Launching the Group By Feature ......................................................................................................... 91
Using the Group By Feature ................................................................................................................ 91
Group By – Basic View ......................................................................................................................... 92
Group By – Advanced View ................................................................................................................. 93
Group By for All Rows ................................................................................................................... 95
Determining the Average Salary per Department ............................................................................... 95
Creating Nested Tables ........................................................................................................................ 96
Calculating the Salary Deviation .......................................................................................................... 98
Unpivot Columns – Basics ............................................................................................................. 99
Unpivot Example.................................................................................................................................. 99
Trimming the Unpivot Fat.................................................................................................................. 101
Unpivot Columns – How to Overcome Common Errors ................................................................. 102
Common Error #1: Adding Data Post-Data Typing ........................................................................... 102
Examining the Unpivot Query ........................................................................................................... 103

Correcting the Unpivot Logic Error .................................................................................................... 104
Common Error #2: Dealing with NULLs in the Source Data ............................................................. 104
Potential Problem for Replacing Nulls with Zeroes ........................................................................... 106
Pivot Column – Create Readable Reports Fast ............................................................................. 108
Creating a Pivot Report ...................................................................................................................... 109
Split Column – The Problem That’s Easy to Miss .......................................................................... 111
The Problem....................................................................................................................................... 111
The Solution ....................................................................................................................................... 111
The Flaw In the Logic ......................................................................................................................... 114
The Fix for the Flaw ........................................................................................................................... 114
Split Column by Delimiter – Rows Instead of Columns .................................................................. 115
Splitting the Colors Across Rows ....................................................................................................... 115
Tweaking the List ............................................................................................................................... 117
“Future-Proofing” the Query ............................................................................................................. 118
Part 5
Date and Time Transformations
Date and Time Transformations .................................................................................................. 121
Making the Complex Simple .............................................................................................................. 122
Useful Date Features .................................................................................................................. 124
Date Calculation Examples ................................................................................................................ 125
Useful Time Features .................................................................................................................. 126
Time Calculation Examples ................................................................................................................ 127
Useful Duration Features ............................................................................................................ 128
Duration Calculation Examples .......................................................................................................... 129
Part 6
Custom Columns
Why Use Custom Columns........................................................................................................... 131
Your Imagination has No Limits ......................................................................................................... 131
Custom Columns – Type Compatibility & Date Function ............................................................... 133
Data Type Compatibility .................................................................................................................... 133
Data Type Conversions using Functions ............................................................................................ 133
Date Function .................................................................................................................................... 134

Other Intrinsic Functions ................................................................................................................... 135

Part 7
Power Query Online Data Sources
Connecting to Different Sources .................................................................................................. 137
The Vast World of Connectors........................................................................................................... 137
Import Data from a Website ....................................................................................................... 138
Connecting to a Website ................................................................................................................... 138
Connecting to an Embedded Excel File or Text File on a Webpage .................................................. 139
Updating Connection Credentials...................................................................................................... 139
Import Data from ODATA (Open Data Protocol) .......................................................................... 140
Connecting to ODATA ........................................................................................................................ 140
Get Google Sheet Data with Power Query ................................................................................... 141
Obtaining the Google Sheets URL...................................................................................................... 141
Connecting to the Google Sheets URL ............................................................................................... 141
Connect to Microsoft Exchange – Get Outlook Email ................................................................... 143
Extracting Content from Email Messages.......................................................................................... 143
Connect to SharePoint or OneDrive for Business .......................................................................... 145
Extracting Content from Cloud-based Storage.................................................................................. 145
Part 8
Combining & Appending Data
Why Append Data? The Difference Between Append and Merge ................................................ 149
Appending Data ................................................................................................................................. 149
Creating the Appended Query ........................................................................................................... 150
Merging Data ..................................................................................................................................... 151
Creating the Merged Query ............................................................................................................... 151
Part 9
Merge Options Join Kinds Explained
Overview of Merge Options – Understanding Join Kinds .............................................................. 155
Advantages of Merges ....................................................................................................................... 155
Merge Options ................................................................................................................................... 155
Left vs. Right / First vs. Second .......................................................................................................... 156
Understanding Join Kinds .................................................................................................................. 157
Merge Based on Multiple Columns .............................................................................................. 158

Creating a Compound Key ................................................................................................................. 158

Merge to Get Multiple Match Results.......................................................................................... 160
Dealing with One-t-Many Results ...................................................................................................... 160
Retrieving the Match with the Latest Date ....................................................................................... 161
Merging Text Columns – Dangers & Pitfalls ................................................................................. 164
It’s the Little Things that Get Us ........................................................................................................ 164
How to Use Fuzzy Match ............................................................................................................. 165
When Close is Good Enough .............................................................................................................. 165
Fuzzy Match Options ......................................................................................................................... 166
Using a Transformation Table............................................................................................................ 166
Part 10
When to use Power Pivot & Load to Data Model
When to Load to the Data Model ................................................................................................ 169
Out of Sight, but Not Out of Mind ..................................................................................................... 169
Power Pivot and the Data Model ...................................................................................................... 169
Adding Query Results to the Data Model .......................................................................................... 169
When to Use the Data Model ............................................................................................................ 170
Is Power Query Needed to Load Data into the Data Model.............................................................. 170
Part 11
Understanding M Formula Language
M Language – How M Thinks (let Expression & Values) ............................................................... 173
Viewing Code in the Advanced Editor ............................................................................................... 173
Expressions ........................................................................................................................................ 174
The let Expression .............................................................................................................................. 174
Storing the Individual Step Results .................................................................................................... 175
The in Expression ............................................................................................................................... 175
Comments in M Code ........................................................................................................................ 176
Step Name Syntax .............................................................................................................................. 176
“M” Characteristics ............................................................................................................................ 176
Defining & Invoking Custom M Functions .................................................................................... 178
Function Syntax ................................................................................................................................. 178
Multi-Step Functions ......................................................................................................................... 180
Saving a Function ............................................................................................................................... 180

Using a Function in a Query ............................................................................................................... 180

Updating Function Logic .................................................................................................................... 182
Embedded Functions Within Queries ................................................................................................ 182
Reference Guide for Standard M Functions.................................................................................. 183
Learn About Power Query Formulas ................................................................................................. 183
Built-In Function Documentation ...................................................................................................... 186
Function Context ............................................................................................................................... 187
Alternate Method for Help on a Specific Function ............................................................................ 188
Tables, Lists, & Records – How to Reference Them in a Table ....................................................... 189
Lists .................................................................................................................................................... 189
Creating a List from a Table ............................................................................................................... 189
Referencing Column Headers in Lists ................................................................................................ 189
Records .............................................................................................................................................. 190
Referencing Tables, Lists, Records, and Cells .................................................................................... 190
Performing a Search on a Table......................................................................................................... 192
Returning a Value from a Searched for Record ................................................................................. 193
Returning a Table from a Table ......................................................................................................... 193
Brackets & Lookup Operators in M Code ..................................................................................... 194
3 Important Power Query Rules ........................................................................................................ 195
Creating Lists, Tables, & Records – to use as Parameters or Testing of Functions ......................... 196
Creating a List from Scratch ............................................................................................................... 196
Creating a Record from Scratch ......................................................................................................... 199
Creating a Table from Scratch ........................................................................................................... 199
Understanding the Each Keyword – and the Purpose of the Underscore....................................... 200
Shorthand Notation ........................................................................................................................... 200
Underscore Notation/Reference ....................................................................................................... 200
Using Power Query Parameters .................................................................................................. 202
Creating Managed Parameters .......................................................................................................... 203
Update the Query to Use the Parameter .......................................................................................... 204
Using the Parameter .......................................................................................................................... 204
Speed Up Queries – Table.Buffer & How to Test Impact ............................................................... 206
Why Is This Behavior Not the Default?.............................................................................................. 207
Testing Performance On Large Data Sets .......................................................................................... 207
Counting the Rows in a Data Set ....................................................................................................... 207
Query Folding – Improve Performance for Relational Databases ................................................. 209

What is Query Folding? ..................................................................................................................... 209
Viewing Query Folding Instructions................................................................................................... 209
Breaking Query Folding ..................................................................................................................... 210
IF Then Statement – Lookup Operators to Get Value from Previous Row ..................................... 211
Creating Row Against Row Comparisons........................................................................................... 211
Error Handling in Power Query – Bulk Replace Lookup with Try Otherwise ................................. 213
Testing the Waters............................................................................................................................. 213
The Purpose of Try Otherwise ........................................................................................................... 213
Part 12
Working with Lists & Table Function
Power Query Text Functions........................................................................................................ 217
Text.Contains, Text.Replace, etc. ...................................................................................................... 217
Text.PositionOf .................................................................................................................................. 217
More Resources
Learn More
More Resources .......................................................................................................................... 222

Power Query Reference Book

This Power Query Reference Guide is accompanying documentation for my online Master Excel Power
Query: Beginner to Advanced (including M) course on XelPlus.com/courses. Please do not reproduce or
transmit in any form without permission.
We (XelPlus e.U.) have taken every effort to ensure the accuracy of this manual. In case you discover
any discrepancies, please send us a quick email to info@XelPlus.com.

Availability of Power Query

Power Query is natively integrated in several Microsoft products, including the following.
Microsoft Power Query enables Excel users to import data from a wide range of data sources
Excel into Excel for analytics and visualizations.
Starting with Excel 2016, Power Query capabilities are natively integrated and can be
found under the “Get & Transform” section of the Data tab in the Excel Desktop
ribbon.
Excel 2010 and 2013 users can also leverage Power Query by installing the Microsoft
Power Query for Excel add-in.
Microsoft Power Query enables data analysts and report authors to connect and transform data
Power BI as part of creating Power BI reports using Power BI Desktop.
Microsoft SQL Business Intelligence Developers can create Azure Analysis Services and SQL Server
Server Data Analysis Services tabular models using SQL Server Data Tools for Visual Studio. Within
Tools for this experience, users can leverage Power Query to access and reshape data as part
Visual Studio of defining tabular models.
Common Common Data Service lets you securely store and manage data that's used by
Data Service business applications. Data within Common Data Service is stored within a set of
entities. An entity is a set of records used to store data, similar to how a table stores
data within a database.
Common Data Service includes a base set of standard entities that cover typical
scenarios, but you can also create custom entities specific to your organization and
populate them with data using Power Query. App makers can then use Power Apps
to build rich applications using this data.

About Leila Gharani

Leila Gharani is a Microsoft MVP & a bestselling online course instructor. She runs XelPlus.com an
spreadsheet resource site to help people gain the knowledge they need so they can create useful tools,
solve problems and get more done.
Her background is Masters in Economics, Economist, Consultant, Oracle HFM Accounting Systems Expert
& Project Manager. Find out more here.

The Accompanying Files

The examples shown in this manual are taken from the Excel files available inside the online course.
All files are available as a zip file.

Part 1
Power Query
Basics
Power Query Power Query Basics
POWER QUERY EDITOR
Overview
You can open the Power Query editor from the Excel’s Data tab, by going to Data > Get Data > Launch
Power Query Editor.
You can also automatically launch the editor by importing data into Power Query. Simply select the
appropriate option from the Get & Transform Data Group on the Ribbon.
Copyright © 2020 Leila Gharani 3|Page

Updating the Source Location/Name

1. Open the query in Power Query
2. Update the file location/name reference by clicking the gear icon next to the Source step and
edit the location/name
The advantage of the gear icon is that you are presented with a user-friendly way to browse to the new
location, thus avoiding typographic errors.
NOTE: Any time you see a gear icon next to a step, it means that the step has parameters that can be
adjusted.
These adjustments are usually presented in user-friendly dialog boxes, eliminating the need to write
complex formulas and statements.
4|Page Copyright © 2020 Leila Gharani

Deleting a Query Step

Power Query does not possess an Undo feature. If you perform an undesirable transformation on the
data, click the that appears to the left of a step name when you hover your cursor over the step.
Be aware of the following behaviors when deleting a query step:
• Deleting a step cannot be undone

• Deleting a step may “break” following steps that were dependent on the deleted step
Bulk Step Deletion
If you have a list of steps and you wish to delete all steps after a specific step, instead of deleting each
step individually, you can save time by right-clicking the first step to delete and select Delete Until End.
This will delete all steps from that point and after in the step list.

Quicklist for Common Table Transformations

If you right-click the button located to the left of the first column header, you can gain access to the
quick list for table transformations.
Many of the most common table transformation tools are in a single, easy to navigate list.
Transform Versus Add Column

The tabs labeled Transform and Add Column have many of the same features. This can be confusing for
beginners as to which is the correct feature version to use.
• Transform will replace the original data (column) with the transformed version of the data.
• Add Column will create a new column to hold the results of the transformation.

Split Text by Delimiter

You can access the text splitting feature multiple ways:
• Home (tab) → Transform (group) → Split Column

• Transform (tab) → Text Column (group) → Split Column
• Right-click a column heading → Split Column
The Split Column feature has several variations by which the split will be performed. Many of these will
be discussed in later sections.
The most widely used option is By Delimiter.
From here, you can select the delimiter and whether you want to split at every occurrence or just the
first or last occurrence.
Automatic Data Typing

Power Query will automatically detect the data types when certain events occur:
• A row is promoted to a header row

• A column is split into multiple columns
If this step was not desired, you can click the that appears to the left of the change type step to
delete the step.
If this automatic data typing feature is unwanted, you can deactivate the feature at the file-level or the
program level.
File-Level Deactivation
File → Options & Settings → Query Options → Current Workbook → Data Load → Type Detection →
Detect column types and headers for unstructured sources.
Program-Level Deactivation
File → Options & Settings → Query Options → Global → Data Load → Type Detection → Never detect
column types and headers for unstructured sources.
Rename a Column
Columns can be renamed in several ways:
• Transform (tab) → Any Column (group) → Rename

• Right-click a column heading and select Rename
• Double-click a column heading to enter rename mode
Removing (Deleting) a Column

Columns can be deleted (known as “vertical filtering”) several ways:
• Home (tab) → Manage Columns (group) → Remove Columns

• Right-click a column heading and select Remove
• Click a column heading and press Delete on the keyboard
NOTE: If you have many columns to delete and only a few columns to preserve, it may be more efficient
to select the columns you wish to keep and use the Remove Other Columns feature to delete the
unwanted columns.
Choosing Columns to Keep and Columns to Delete

To select or deselect multiple columns, a selectable list of columns can be displayed by clicking Home
(tab) → Manage Columns (group) → Choose Columns.

This is a superior way of removing unwanted columns, as opposed to selecting unwanted columns and
deleting them.
Later, if you change your mind, you can click the gear icon and reselect a column to add it back into
the data set.
PRO TIP: If you perform the same operation multiple times in a row, Power Query will consolidate all like
steps into a single step in the Applied Steps list. This means, if you delete 5 columns then rename 5
columns, you will only see 2 steps in the Applied Steps list.
Conversely, if you were to delete a column, then rename a column and repeat this 4 more times, you will
see 10 steps in the Applied Steps list.
Perform Quick Mathematical Operations

If you want to perform common mathematical operations such as multiplying the price by quantity or
adding subtotal and tax, you can select the 2 columns of information (using the CTRL key) and select Add
Column (tab) → From Number (tab) → Standard → {operation}.
NOTE: Be mindful of the order at which columns are selected. This is important when performing
operations such as division (ex: Total  Quantity, versus Quantity  Total).
Refreshing Source Data During Query Composition

If you believe the data may have changed since you opened the Power Query Editor, you can update the
preview data by selecting Home (tab) → Query (group) → Refresh Preview.

QUICK INSIGHTS ON DATA – COLUMN PROFILING, DATA QUALITY &

DISTRIBUTION
In an attempt to improve performance when connecting to data sources, Power Query downloads a
maximum of 1,000 rows of data (if you have fewer rows, you will receive the entire data set.)
This is what is known as “the first three inches of the fire hose”. Typically, it’s enough of a
representation of the data to give you a good idea of the entire data set.
This works well for most data sets, but can cause issues with certain tools like filtering, when proposed
lists are incomplete because the full data set has yet to be examined.
Displaying a Complete List When Filtering

If you display a filter dropdown list and there appear to be missing entries, those entries may not exist in
the first 1,000 rows of the data. Power Query will inform you that there may be items in the data that
exist beyond the initial 1,000 entries.
You can force Power Query to read then entire data set for that field by clicking Load More.
This will read the fill data set and display a complete list of items for that column.
10 | P a g e Copyright © 2020 Leila Gharani

Displaying Column Profile Information

To display statistical information about a selected column, click View (tab) → Data Preview (group) →
Column Profile.
When you select a column, statistical and distribution information is displayed for the first 1,000 rows of
data.
This may provide inaccurate information regarding such things as…
• Count of rows
• Empty cells
• Min and max values
Copyright © 2020 Leila Gharani 11 | P a g e

Displaying a Complete List for Selected Column

To display accurate statistics about the entire data set, force Power Query to load the entire data set by
clicking in the lower-left corner where it says “Column profiling based on top 1000 rows” and change
the option to “Column profile based on entire data set”.
NOTE: System performance may suffer when loading the entire data set. Consider loading all data to
capture some quick insights into the data, but return the captured set to the first 1,000 to improve
performance.
Column Distribution
To view the number of distinct versus unique values in each column, click View (tab) → Data Preview
(group) → Column Distribution.
This provides a miniature column chart at the top of each column to display the relationship between
distinct and unique values.
NOTE: These mini-charts are based on the setting for “first 1,000” or “all records”.
Column Quality
To display the percentage of valid data vs. errors vs. empty cells, click View (tab) → Data Preview
(group) → Column Quality.
The column headings now display percentages of each category for the column.
NOTE: These percentages are based on the setting for “first 1,000” or “all records”.
Using Monospaced Fonts

If you wish each character to have the same width (no text kerning), select View (tab) → Data Preview
(group) → Monospaced.
This may make the text easier to read, especially when working with columns containing large numbers.

Discover the Total Number of Rows

To quickly discover the number of rows in a data set, select Transform (tab) → Table (group) → Count
Rows.
This will reduce the entire data set to a single value that represents the total number of rows.
This is often performed to satisfy a casual curiosity. The step is often deleted once the question has
been answered.

FORMULA BAR – APPLIED STEPS & M CODE

Power Query does an admirable job of recording all our manipulations in the Applied Steps window.
However, the names provided to the steps are sometimes less than obvious.
Renaming Applied Steps

A better way to document your steps, and make actions more understandable, is to rename the steps.
To rename a step, either click a step name and press F2 or right-click a step name and select Rename.
Other Applied Steps Options

When you right-click a step name, there are other helpful options listed in the menu, such as:
• Delete/ Delete Until End

• Move Up / Move Down
• Insert Step After
• Extract Previous (this one is VERY cool)
Adding Documentation to Applied Steps

Creating a descriptive name for a step helps make queries more understandable, but that only goes so
far due to limited space.
If you need to document more detailed information about a step (i.e. justification for performing the
step or mathematical formula behind the step), right-click the step and select Properties.
In the Description field, write what makes the most sense for your documentation needs.
Power Query Formula Bar

To gain greater insight into what each step is performing, we can activate the Formula Bar by selecting
View (tab) → Layout (group) → Formula Bar.
Similar to Excel’s Formula Bar, we can see the “behind the scenes” instructions for the selected step.
These instructions are written in the Power Query ‘M Language’ (Data Mashup Language).
This is especially useful for steps that do not have the gear icon . We can see in the Formula Bar what
is occurring and potentially perform a direct-edit on the instruction.

For steps that have more instructions than can be viewed on a single line of the Formula Bar, click the
small down-arrow to the right of the Formula Bar to expand the height of the Formula Bar.
Advanced Editor
Although you can see each step individually in the Formula Bar, it’s difficult to remember what you saw
in one step when comparing to another step.
The entire set of “M” code can be displayed by opening the Advanced Editor.
To open the Advanced Editor, select Home (tab) → Query (group) → Advanced Editor or by selecting
View (tab) → Advanced (group) → Advanced Editor.
This will display the entire instruction set of “M” code for the query.

CLOSE & LOAD DESTINATIONS
Working with Existing Queries

When you edit an existing query, the only available option is “Close & Load” which defaults to the
existing destination.
If you need to change the destination, return to Excel and right-click the query in the Queries &
Connections panel and select “Load To…”
This will open the Import Data dialog box where you can select a new destination for the query output.

Creating a Data Connection

A data connection is where you have the steps defined for processing the data, but the data is not
loaded into Excel in any form. The data is “waiting to be called” when needed.
Connection Only queries are good for data that will be used for an intermediate step to aid in
processing, but the results are not needed for the final output.
Connection Only queries greatly reduce file size. This is because the data remains at the source and a
copy is not stored locally in the Excel file.
Changing the Default Load Destination

The default load destination for Power Query is to load to an Excel Table.
If you would rather the default location be to a Connection Only query, or a table and/or the Data
Model, select File (tab) → Options & Settings → Query Options → Global → Data Load → Default
Query Load Settings.
Note: Selecting a custom load but NOT selecting a worksheet or Data Model will establish a Connection
Only query.
“Saving” Your Queries

If you are in the middle of creating a query and you need to leave the query editor, but you don’t want
to discard your work, nor are you ready to load the results into Excel, select “Close & Load to…” and
establish a Connection Only query.
This will allow you to return later and resume work without performing a potentially incomplete data
load.

REFRESH DATA AND REFRESH OPTIONS

When the source data changes, the query results must be refreshed.
There are several ways of performing a refresh operation:
1. Select Data (tab) → Queries & Connections (group) → Refresh All.
2. Right-click on a query results table and select Refresh.
3. Right-click a query in the Queries & Connections panel and select Refresh.
4. Click the Refresh icon in the upper-right corner of the query.
Refresh Options
Each query has a set of refresh options that will allow for a certain degree of automation.
Select Data (tab) → Queries & Connections (group) → Properties to open the Properties dialog box.

Some useful settings include:
• Enable background refresh – this allows you to continue working in Excel as the query is being
refreshed. Disabling this feature is useful when the results of the query are needed for another
operation (like a macro), and you don’t want the following step to run prematurely.
• Refresh every N minutes – this schedules a refresh at a defined interval (in minutes). The
workbook needs to be open for this option to work.
• Refresh data when opening the file – self-explanatory.

Part 2
Important
Power Query
Tips & Tricks
Power Query Important Power Query Tips & Tricks
UPLOADING DATA FROM EXCEL

There are multiple ways to upload data from Excel into Power Query. Two of the most used ways are
from tables and named ranges.
Uploading from Tables

Excel requires that all data sources from Excel must be in the form of a proper Excel Table (when not
working with Named Ranges.) If the table is not already in the form of an Excel Table, Power Query will
automatically “upgrade” the plain table to an Excel Table. Excel will assign a default name, such as
“Table1”, to the table.
If you are creating reports for long-term use or reports that cull data from multiple sources, it’s a good
idea to pre-convert the plain table to an Excel Table and give the table a better name before bringing the
data into Power Query.
1. Select the data range and upgrade to an Excel Table. This can be done via:
• CTRL-T keyboard shortcut
• Home (tab) → Styles (group) → Format as Table
• Insert (tab) → Tables (group) → Table
2. Give the table a proper name, like “SalesTable”, by selecting Table Design (tab) → Properties
(group) → Table Name.
3. If you wish to retain your original table art (i.e. colors, fonts, borders, etc.), remove the
automatically applied table layout by selecting Table Design (tab) → Table Styles (group) →
bottom scroll button → Clear.
Uploading the Data into Power Query
4. To send the newly upgraded data into Power Query, click in the data and select Data (tab) → Get
& Transform Data (group) → From Table/Range.
5. Rename the query output to something like “SalesReport”. This will become the name of the
output table on the Excel worksheet.

The Query Ribbon

Once we have loaded the data back into Excel, we have access to a new ribbon named Query.
The Query ribbon contains many useful features for working with the query as an object, such as:
• Refresh
• Edit
• Delete
• Duplicate / Reference
• Merge / Append
These features are also available by way of right-clicking on a query in the Queries & Connections panel.
Useful Table Settings
Two table settings that are worth exploring:
• Table Design (tab) → External Table Data (group) → Properties → Adjust column width.
• Table Design (tab) → External Table Data (group) → Properties → Preserve cell formatting.
Uploading from Named Ranges

If a data set has been given a name (i.e. Named Range), Power Query will leave the original data “as-is”
and not upgrade the data to an Excel Table.
The downside to this is that Named Ranges are not dynamic. If data is added to the originally defined
Named Range, Power Query is unable to “see” the new data until the user manually updates the Named
Range reference.
Solutions to this problem include:
• Define a range larger than your data to account for future data addition
• Create a dynamic named range by using formulas in the Name Manager to calculate the
height/width of the data range
NOTE: If you are using a Named Range to define your data, select the Named Range before sending it to
Power Query. This prevents Power Query from selecting what it thinks is your data, which may not
match your Named Range definition. This will also prevent Power Query from upgrading your data into
an Excel Table.

Handling Changes to Source

When importing data into Excel using Power Query, a connection to the source material is created. This
way, when the source material changes, you can click “refresh” and the same source is read to import
any additions, deletion, or changes into the report.
The connection is recorded as an absolute address; in other words, the full drive\path\filename is
recorded. If any element of that address changes, the query will break and be unable to perform a
refresh operation.
Other things that can “break” your query include:
• Changes to heading names

• Changes to table names
• Changes to sheet names
Correcting Changes in Location or Filename

If a file’s name or location changes and an update is performed on the Power Query output, we are
presented with an error message.
To correct a query that refers to a file by location and name:

4. Click the Go To Error button to be taken to the offending step, or select the Source step in the
Applied Steps list
5. Update the file location/name reference by either

• Editing the reference in the Formula Bar
• Click the gear icon next to the Source step and edit the location/name

The advantage of the gear icon is that you are presented with a user-friendly way to browse to the new
location, thus avoiding typographic errors.
Correcting Changes to Sheet Names

If a sheet’s name changes and an update is performed on the Power Query output, we are presented
with an error message.
To correct a query step that refers to a sheet name:

2. Click the Edit Settings button to be taken to the offending step, or select the Navigation step in
the Applied Steps list
3. Update the sheet name reference by either

• Editing the sheet name reference in the Formula Bar
• Click the gear icon next to the Navigation step and point to the new sheet name

NOTE: Remember that Power Query is case-sensitive. Ensure you type the names as they appear in the
source material.
Correcting Changes to Table Names

If a table’s name changes and an update is performed on the Power Query output, we are presented
with an error message.
To correct a query step that refers to a table name:

2. Click the Go To Error button to be taken to the offending step, or select the Source step in the
Applied Steps list
3. Update the incorrect table name to the new table name by editing the reference in the Formula
Bar

Correcting Changes to Column Heading Names

If a table’s heading name changes and an update is performed on the Power Query output, we are
presented with an error message.
To correct a query step that refers to a heading name:

We are presented with the following error message.
Unfortunately, we do not have any buttons to lead us to the offending step, nor any gear icon to open a
user-friendly correction window. This is why having the Formula Bar visible is so important.
5. Update the entry in the Formula Bar to display the correct column heading name.
NOTE: Remember that Power Query is case-sensitive. Ensure you type the names as they appear in the
source material.

Correcting Changes Using the Advanced Editor

If you have many changes to perform and you feel confident in your typing accuracy, you can open the
Advanced editor and perform many of the updates in a single operation.
Open the Advanced Editor in Power Query by selecting Home (tab) → Query (group) → Advanced
Editor.
Make the necessary changes in the editor and click when finished.
IMPORTANT: Remember that Power Query does not posses the Undo feature. Consider creating a
backup of the file/query before editing. This way, if you make a mistake during the direct editing of the
query, you can restore your query to its original state.

UNDERSTANDING DATA TYPES
The Importance of Data Types

It is very important to define your fields (columns) with the proper
data types. This ensures the following:
• Data can be properly interpreted/understood

• Data can be manipulated properly
• Data cannot be manipulated improperly
• Proper level if numeric precision is maintained
Data Type Description and Use
Maximum 15 digits regardless of decimal

Decimal Number
placement.
4 digit to the right of decimal; maximum 19
Currency digits regardless of decimal placement.
Anything after the 4th decimal place is rounded.
Integer value (no decimal precision). Maximum 19 digits. Any fractions are
Whole Number
rounded.
Displayed as percentages in Power Query; displayed as decimals in a
Percentage
workbook.
Date/Time Date & Time in a single column. Stored as a decimal type.
Date Dates from 1900 to 9999 are supported. Stored as whole numbers.
Time Time only. Stored as decimal type.
Date/Time/Timezone UTC Date/Time
Length of time shown as days, hours, minutes, and seconds. Stored as

Duration
decimal type.
Text Text and numbers. Maximum of 268K characters.
True/False Boolean value.
Binary A sequence of bytes (e.g. when loading files from a folder).
Important if importing data from sources that have different regional

Using Locale…
settings.

DATA TYPES VS. FORMATTING (AND NULL FORMATTING)

The fundamental principles behind storing versus viewing numbers in Excel and Power Query are:
In Excel
• numbers are calculated as stored, not as displayed
• the formatting of numbers does not affect the precision of numbers
In Power Query
• formatting numbers changes the stored value
• once a number is formatted to a lesser accuracy, the original level of accuracy cannot be restored
without deleting prior query steps
Excel Examples
If we take a value in Excel that has 2-decimal place precision and we format the value as a whole
number, the number displayed will be rounded but the underlying value remains unchanged.
Power Query Examples

Reducing Decimal Precision
If we were to load into Power Query a column of values with 2-decimal place precision and we then set
the data type to Whole Number, the fractional side of the number is discarded and the whole number
side is rounded.

If we examine the number more closely, we can see that the change in data type has altered the value.
This precision cannot be restored once the alteration has taken place. It would be necessary to delete
the step that applied the change to regain the original precision.
Increasing Decimal Precision
If you import whole numbers into Power Query and later add data that has fractional precision, you will
lose the fractions.
This is because the original numbers are data typed as Whole Numbers. Future values will be converted
to Whole Numbers, thus losing their fractional precision.
If you believe that you may encounter fractions in your data, it’s a good idea to set the data type to
Decimal as a way of “future-proofing” your query.
Currency-based Values
When values are data typed as Currency in Power Query, the results are displayed in the table with 2-
decimal place precision, but the underlying values are stored and calculated at 4-decimal place
precision.

Invoking Automatic Data Types

If you have removed the originally established data types and want to let Power Query “figure it out”,
select a column, separate columns using CTRL, or all columns using CTRL-A, and click Transform (tab) →
Any Column (group) → Detect Data Type.
Working with NULLs

If a cell is empty when brought into Power Query, the corresponding cell will be displayed with a NULL
indicator.
This is an indication that the cell is truly empty; no spaces or hidden characters

POWER QUERY SHORTCUTS

As anyone who has worked with Excel, or any other application will tell you, learning time-saving
shortcuts is one of the fastest ways to shorten the duration of your workflow.
Below are some of the more useful Power Query shortcuts. This is by no means a comprehensive list,
but it’s a great place to start for reducing work.
Shortcut Task
F2 Edit the name of a column or query
Arrow keys (L & R) Navigate left or right through columns
CTRL key Select multiple, non-contiguous columns
Shift key Select contiguous columns
CTRL-A Select ALL columns
CTRL-Space Select the entire column of a selected cell
ALT (while opening Excel) Open a second, unrelated instance of Excel
Common time-saving features

Selecting / Deselecting Columns
To select or deselect multiple columns, a selectable list of columns can be displayed by clicking Home
(tab) → Manage Columns (group) → Choose Columns.

This is a superior way of removing unwanted columns, as opposed to selecting unwanted columns and
deleting them.
Later, if you change your mind, you can click the gear icon and reselect a column to add it back into
the data set.
Detecting Data Types
It’s not uncommon to delete the automatically applied type detection step from the Applied Steps list in
Power Query.
When it comes time to perform data type detection, you can select the column(s) you want to data type
and click Transform (tab) → Any Column (group) → Detect Data Type.

ERROR HANDLING – FINDING & CORRECTING ERRORS IN DATA

We’ve seen how to handle common query-wide errors, such as:
• when the source files location or name changes

• when the sheet name changes
• when the table name changes
Now we will demonstrate common techniques for handling errors within columns of data.
Errors that should be corrected at the source level

Certain errors should be corrected in the data before loading into Power Query. Common issues
include:
• A mixture of date formats from differing regions

• Column name inconsistencies when appending multiple files in a folder
Determining which rows contain errors

If your query returns a message stating that the results contain errors, click the error count to generate a
detailed list of data rows that contain errors.
Power Query creates a new query that extracts the complete record for all rows of the results query that
contain errors. Each row is prefaced with a row number indicating the record position in the source
data.

If you click to the right of the error message, Power Query displays a detailed explanation of the error.
Errors to correct in Power Query

Many errors occur in Power Query due to an incorrectly applied data type.
In other words, if a column contains sales and certain records contain messages such as “No Sale”.
When the column has a Currency data type applied to it, the cells holding text will display an error due
to a data type mismatch.

Methods to Deal With Errors in Power Query

Remove Rows with Errors
If the rows containing errors in each column are not needed, you can delete the error rows by selecting
the column containing errors and click Home (tab) → Reduce Rows (group) → Remove Rows →
Remove Errors.
Replace Errors with Meaningful Data

You can replace the errors with meaningful data, such as a default date, zeroes, nulls, etc.
Select the column containing errors and click Transform (tab) → Any Column (group) → Replace Values
→ Replace Errors.
In the Replace Errors dialog box, enter the value you wish to replace the errors with, such as 0 (zero).
The results are as follows.

MORE DATA VIEWS – DUPLICATE OR REFERENCE QUERY?

When working with Power Query, you are likely to encounter two familiar scenarios:
1. Creating multiple query outputs based on the same data set.
2. Large queries that you wish to break down into sections for ease of understanding, maintenance,
and reusability.
Duplicating a Query
Creates a second copy of your existing query where the actions and results are independent of the
original query. This is like copying a file. The two queries have no connection between them; altering
one does not affect the other.
Referencing a Query
Creates a new query that is dependent on the original query. The output of the original query becomes
the input for the dependent query.
How to Duplicate a Query

There are several ways you can duplicate a query, depending on where you are at the moment.
If you are in Excel:
• From the Queries & Connections panel, right-click a query and select Duplicate.
• Select a query from the Queries & Connections panel, then select Query (tab) → Reuse (group)
→ Duplicate.

If you are in Power Query:

• From the Queries list, right-click a query and select Duplicate.
• Select a query from the Queries list, then select Home (tab) → Query (group) → Manage →
Duplicate.
How to Reference a Query

There are several ways you can reference a query, depending on where you are at the moment.
• From the Queries & Connections panel, right-click a query and select Reference.

• Select a query from the Queries & Connections panel, then select Query (tab) → Reuse (group)
→ Reference.

• From the Queries list, right-click a query and select Reference.
Reference.

QUERY DEPENDENCY – VISUALIZE QUERY RELATIONS
Viewing Query Dependencies Graphically

As seen in the previous section, queries can reference other queries.
When a query uses the output of another query as its input, a dependency is created. The input query is
dependent on the output query.
A query that has dependent queries cannot be deleted without first deleting the dependent queries.
If the output query were to be deleted, the dependent queries would no longer have a source with
which to draw data.
To visualize these dependencies in Power Query, select View (tab) → Dependencies (group) → Query
Dependencies.
The following Query Dependencies dialog box will graphically display the queries, the relationship
hierarchy, and their load destinations.

QUERY MANAGEMENT – DELETE, MANAGE, COPY QUERIES & BACKUP

RESULTS
Delete a Query
• From the Queries & Connections panel, right-click a query and select Delete or press the Delete
key on the keyboard.
• Select a query from the Queries & Connections panel, then select Query (tab) → Edit (group) →
Delete.

• From the Queries list, right-click a query and select Delete.

Delete.
NOTE: If you delete a query, the output of the query remains in the workbook.
Deleting All Queries at Once

Suppose you want to send the query output results to a client but you don’t want them to see how the
data was collected and processed. If you have several queries and you wish to delete all queries in a
workbook, perform the following steps:
1. Consider saving the file under a different name. This way you will retain a version of the file with
the queries.
2. In the version of the file to clean, select File (tab) → Info → Check for Issues → Inspect
Document.
3. In the Document Inspector dialog box, click Inspect.
4. Scroll down to the category labeled Custom XML Data and click Remove All.
5. Click Close.
All the queries will have been removed while leaving the query results intact.
Copy a Query to Another Workbook

When you want to copy an entire query to another workbook, it’s a simple matter of Copy/Paste.
1. From the Queries & Connections panel, right-click a query and select Copy.
2. Start a new workbook or open an existing workbook.
3. Open the queries list by selecting Data (tab) → Queries & Connections (group) → Queries &
Connections.
4. From the Queries & Connections panel, right-click in an empty part of the panel and select Paste.
Export a Query for Later Use

You can export a query to a file that you can import at a later date, or send it to someone for them to
import into their workbook.
1. From the Queries & Connections panel, right-click a query and select Export Connection File.
2. Save the query as an .ODC (Open Database Connection) file.
NOTE: The .ODC file is an eXtensible Markup Language (XML) file that can be viewed in any text editor.
3. Start a new workbook or open an existing workbook.

4. Select Data (tab) → Get & Transform Data (group) → Existing Connections.

5. In the Existing Connections dialog box, select the applicable query from the list.
NOTE: If the query is being transferred to a different computer, you can click “Browse for more…” and
manually locate the .ODC file.
Organizing Queries
If you are working in a workbook with many queries (perhaps dozens), you can organize your queries
into Query Groups based on similar purpose, function, or phase.
Create a Query Group
1. From the Queries & Connections panel, click a query then hold the CTRL key and select one or
more other queries.
2. Right-click one of the selected queries and click Move to Group.
3. You can select one of the existing groups or create a new group by selecting “New Group…” If
you are creating a new group, give the group a name and consider adding in the Description field
some explanation as to the purpose of the queries in this group.

The results can be expanded or collapsed to create a cleaner appearance to the queries list.
Remove a Query Group

To remove a query group, right-click the query group and click Ungroup.
All ungrouped queries are placed in the “Other Queries” group.

Part 3
Helpful
Power Query
Transformations
Power Query Helpful Power Query Transformations
TEXT TRANSFORMATIONS – FORMAT, EXTRACT & MORE
Transformation #1: Split Text by Delimiter

Our first transformation is to separate the Department and the Position into 2 columns.
Because the Department and Position are separated by “space – forward slash – space” characters, we
can leverage these characters as a delimiter to assist in the separation process.
1. Click in the table and select Data (tab) -> Get & Transform (group) -> From Table/Range.
2. Rename the transformation to “ProperData”.
3. Select the Department/Position column and click Home (tab) -> Transform (group) -> Split
Column -> By Delimiter.

4. In the Split Column by Delimiter dialog box, select Custom and use a forward slash (/) as the
delimiter.
5. Click OK.
NOTES:
• Once the split has been performed, another “Changed Type” step is added. This is to reassess
the newly split data for possible new data types. We could delete this step as it is not creating
anything we didn’t already possess.
• Technically, we could have used a “ / “ (space, forward slash, space) as the custom delimiter.
This would have allowed us to eliminate the upcoming “Trim” step.
Removing the Leading & Trailing Spaces
If you select to the right of one of the Department names, you will be presented with a preview window
at the bottom on the screen.
If you then click to the right of the data in the preview window you will notice that the cursor does not
rest directly next to the last letter; there is a trailing space.

We can also discover that there is a leading space before the Position in the second column.
Trimming the Data

To remove the trailing and leading spaces, select the Department column, hold CTRL and select the
Position column. Select Transform (tab) -> Text Column (group) -> Format -> Trim.
I prefer using the “space – forward slash – space” approach as it reduces the query by a step.
NOTE: We will deal with the column headings at the end. For now, we can leave them a bit messy.

Transformation #2: Extract Employee ID from Unique Code

Our next step involves the extraction of the Employee ID from within the Unique Code column. Notice
that the format is “last name – employee ID – first name”.
Select the Unique Code column and click Transform (tab) -> Text Column (group) -> Split Column -> By
Non-Digit to Digit.
This separates the Last Name from the Employee ID/First Name.

Next, select the column with Employee ID and Last Name and click Transform (tab) -> Text Column
(group) -> Split Column -> By Digit to Non-Digit.
This separates the Employee ID from the First Name.
Getting rid of the unneeded bits

As we only need the Employee ID, we will remove the newly separated First Name and Last Name
columns.
Click the column of Last Names, then press CTRL and click the column of First Names.
Press the Delete key to remove the selected columns.
Formatting the Employee ID
Remember, one of the requirements was to format the Employee IDs with an “E-“ prefix.

Select the column that contains Employee IDs and click Transform (tab) -> Text Column (group) ->
Format -> Add Prefix.
In the Prefix dialog box, enter a Value of “E-“ and click OK.
We now have our properly formatted Employee IDs.

Transformation #3: Separate Names and Format Casing

Our final set of transformations is to separate the contents of the Full Name column into a First Name
column and Last Name column whilst ignoring any middle name information.
There is a wealth of options sitting withing the Extract feature in Power Query.
We don’t want to transform what we have into a new set of data; we want to leave the original Full
Name column while adding additional columns for First Name and Last Name.
For this operation, we will use the version of Extract located on the Add Column ribbon.
Extracting First Names
Select the Full Name column and click Add Column (tab) -> From Text (group) -> Extract -> Text Before
Delimiter.

In the Text Before Delimiter dialog box, enter a space in the Delimiter field and click OK.
We are presented with a new column that contains all text up to the first space in the Full Name text.
Extracting Last Names

Select the Full Name column and click Add Column (tab) -> From Text (group) -> Extract -> Text After
Delimiter.

In the Text After Delimiter dialog box, enter a space in the Delimiter field. Expand the Advanced
Options and set the Scan for the Delimiter option to “From the end of the input” and click OK.
Starting from the end and “looking” backward is necessary because of the existence of middle names in
some of our records.
If we began our search for a space from left-to-right, we would stop before a middle name and extract
the middle and last names.
We are presented with a new column that contains all text after the last space in the Full Name text.

Format the Names with Proper Casing

The next step is to format the newly added First Name and Last Name columns so that the first letter is
upper-case while the remaining letters are lower-case.
Select the First Name and Last Name columns then click Transform (tab) -> Text Column (group) ->
Format -> Capitalize Each Word.
We now have our properly formatted names.

Finishing Touches
• We no longer require the original column of Full Names, so we can select the Full Name column
and press Delete.
• Rename the column headings with more meaningful names.
Sending the Results to Excel

To send the transformation results back to Excel as a finished table, select Home (tab) → Close (group)
→ Close & Load (lower part of the button) → Close & Load to…
You can load the results to a table on a new sheet, or an existing sheet.

All three requirements have been satisfied and we didn’t have to write a single formula to get the job
done.

MERGE COLUMNS – WHAT TO WATCH OUT FOR

Two very important first steps will govern your success with the Merge Columns feature:
• The order by which you select the columns to be merged.

• The tab you invoke the Merge Columns feature from; Transform or Add Column.
Selecting the Columns to Be Merged

The order in which you select the columns determines the merge order.
This is especially useful because if your data is not in the proper order, you are not required to establish
the order before the merge. You can select the columns based on the order you want the final result,
then perform the merge. The Merge Columns feature will accomplish two tasks in a single step.
Use the CTRL key to select multiple columns.
Launch the Merge Columns feature in one of the following manners:
• To replace the existing content, select Transform (tab) → Text Column (group) → Merge
Columns or right-click the selected columns and select Merge Columns.
• To create additional content, select Add Column (tab) → Text Column (group) → Merge
Columns.

In the Merge Columns dialog box, select the separator (delimiter) you wish to use to separate each
column’s data within the result.
In the New Column Name field, type the name you wish to use for the heating of the new column.
Potential Issue
Because we were including middle names in the above example, names that do not contain a middle
name are padded with two spaces between the first and last name.
Merging Without Extraneous Spaces

The key to getting the proper result (names without extra spaces) is to invoke the Merge Columns step
from the Add Columns tab. This will produce a merged result without extra spaces. Worst Case
Scenario, you may need to add an extra step to delete the original columns if you were intending to
replace them.

Capturing the First Letter Only In the Middle Name

Suppose you only wanted to keep the first letter of the middle name. Before the merge operation:
1. Select the Middle Name column.
2. Select Transform (tab) → Text Column (group) → Extract → First Characters.
3. In the Extract First Characters dialog box, enter a 1 in the Count field and click OK.
Standardizing Middle Initial Casing & Style

If you want to “future-proof” your query to account for times when data may arrive in lower-case
formatting, perform the following step:
1. Select the Middle Name column.
2. Select Transform (tab) → Text Column (group) → Format → UPPERCASE.
4. If you wish to have a period following the Middle Name, select the Middle Name column.
5. Select Transform (tab) → Text Column (group) → Format → Add Suffix.
6. In the Suffix dialog box, enter a “.” (period – no quotes) in the Value field and click OK.

FILL & REPLACE VALUES – TO CREATE A PROPER DATASET

It’s not uncommon to receive a table of information that does not adhere to proper table rules. One
common rule violation is to suppress repetitive items.
Repeat Cells to Adjacent Cells (Fill)

Select the columns you wish to repeat data to adjacent, empty cells and click Transform (tab) → Any
Column (group) → Fill → Down.
Alternately, you can right-click the selected column and select Fill → Down.
Beware of “Not Really Empty” Cells
If you look closely at our Position column, some of the empty cells say null while other cells are blank.
In this case, it is because some of the cells contain a “space” character.

We would need to replace the cells that contain only a space with nulls, while not replacing the space
characters in cells that contain text.

Click in the desired column and select Home (tab) → Transform (group) → Replace Values. In the
Replace Value dialog box, enter a “space” character in the Value to Find field and a “null” (no quotes) in
the Replace With field.
Replacing Data
Do not forget; Power Query is CASE-SENSITIVE. If you were to replace data by keyword, you need to
ensure that all versions of the desired data match the defined casing in the replacement procedure.
To replace text, select the needed column(s) and click Transform (tab) → Any Column (group) →
Replace Values. In the Replace Value dialog box, enter the text you are searching for in the Value to
Find field and the replacement text in the Replace With field.

Advanced Options for Replacing Data
If you click “Advanced Options” in the Replace Values dialog box, you can activate features such as:
• Full cell content matching

• Adding special characters, such as tabs, carriage returns, and non-breaking spaces.
NOTE: The Advanced Options do not contain an option for case-insensitivity. If you need to replace it in
a case-insensitive fashion, you will need to apply a formatting transformation, before the replacement,
that will standardize the casing of the text.
Standardizing Casing
To standardize the casing of data in a column, select the needed column(s) and click Transform (tab) →
Text Column (group) → Format → then the desired casing style (i.e. “Capitalize Each Word”.)
Replacing NULL Values

If you have a column that contains NULL values, and you want to replace them with default text or
default values, select the needed column(s) and click Transform (tab) → Any Column (group) →
Replace Values.
In the Replace Value dialog box, enter “null” (no quotes) in the Value to Find field and the default
text/value in the Replace With field.
SORT DATA – INCLUDING MULTIPLE LEVELS
Simple Sorts
To perform a simple sort (defined as sorting the table by a single column), select the desired column and
click Home (tab) → Sort (group) → Sort Ascending/Sort Descending.
You can also click the Sort/Filter dropdown arrow at the top of the column and select Sort Ascending or
Sort Descending.
Look closely and you can see an “up arrow” indicator in the Sort/Filter button to indicate that the
column is sorted with an ascending option.
Multi-Level Sorts
We have already covered creating a simple sort, so we will use this existing sort and build off it.
The original sort (ascending by Department) will be our Primary sort. Within each Department, we
want to perform a Secondary sort by Position. Perform an ascending sort on the Position column.
Finally, we can sort within each Position by Salary in a descending strategy. This will be our Tertiary
sort. Perform a descending sort by Salary.

Observe the Sort/Filter buttons on the header row.
When you have more than one level in the sort logic, you receive numeric indicators next to the
Sort/Filter buttons to indicate Primary-Secondary-Tertiary order. The numbers are very small, so you
may need to look closely.
Returning to an Original Sort Order

This one is a bit of a sneaky trick.
If you need to create a series of transformations that alter your sort order, but later you need to return
to the original sort order, we can create an Index Column to keep track of the original order.
Before any transformations that will impact sort order, select Add Column (tab) → General (group) →
Index Column.
This creates a column of numbers that we can use as markers to denote the original row number
locations.
Once our transformations are complete, we can perform a simple sort on the Index Column to return
the list to its original order. The Index Column can then be removed.
REMOVE DUPLICATES – INCLUDING MULTIPLE COLUMNS
Remove Duplicates Based on a Single Column

To remove the duplicates in a table based on a single column, select the needed column and click Home
(tab) → Reduce Rows (group) → Remove Rows → Remove Duplicates.
NOTE: You can also KEEP only the duplicate rows by selecting Home (tab) → Reduce Rows (group) →
Keep Rows → Keep Duplicates.
REMEMBER: Power Query is case-sensitive. If you have the same word occurring in your column, but in
differing cases, they will be treated as separate, unique items. Consider standardizing your casing before
the Remove Duplicates operation.
Remove Duplicates Based on Multiple Columns

Removing rows based on multiple columns is identical to a single-column duplicate removal, but with
one added step.
Before the removal operation, select all columns (using CTRL) that will contribute to the criteria (ex:
Department and Position).

NUMBER TRANSFORMATIONS – WHAT TO WATCH OUT FOR

Many useful, built-in mathematical transformations exist in Power Query.
By selecting a column of numbers, we can access these transformations by selecting Transform (tab) →
Number Column (group). Here we see the major categories of transformations.
Adding a Set Value to All Values in a Column

Select the column of values and click Transform (tab) → Number Column (group) → Standard → Add.
In the Add dialog box, in the Value field, enter the value to be added to all values in the column and click
OK.
Rounding Values in a Column (with little know feature)

When working with fractions, you can round the values to a set decimal precision by selecting the
desired column(s) and clicking Transform (tab) → Number Column (group) → Rounding → Round.

In the Round dialog box, in the Decimal Places field, enter the value that denotes the level of decimal
precision and click OK.
Awesome Little Know Ability of Rounding

If you enter a negative value in the Decimal Places field, the rounding will occur to the left of the
decimal place. This means values like 634984, when rounded to a -4 would transform into 640000.
Produce an Aggregation Using Two Columns
NOTE: The following features are only available on the Add Column tab. The options are grayed-out on
the Transform tab. If you do not wish to retain the original columns post-aggregation, you will need to
manually remove the original columns.
If you wish to add the values of two columns to produce a third aggregation column (i.e. add, subtract,
multiply, divide, max, min, median, etc.), Select the two columns and click Transform (tab) → Number
Column (group) → then the appropriate aggregation option.
NOTE: When performing certain mathematical operations (ex: division), the order you select the columns
determines certain argument types, like numerator and denominator.
Adding Columns That Contain Nulls

When adding two columns where some of the rows contain null entries, the null will act as a “poison
pill” which nullifies the Add operation.
There are many ways to deal with this problem. A couple of easy solutions are:
• Replace the null entries with zeroes before the addition.

• Use the Add Column (tab) → From Number (group)→ Statistics → Sum feature instead of the
Add Column (tab) → From Number (group)→ Standard → Add feature.
This behavior is the same when performing other arithmetic operations, like multiplication and
subtraction.
Multiply a Percentage Against a Value

To multiply two columns of values against one another, select the desired columns, click Add Column
(tab) → From Number (group)→ Standard → Multiple.
Remember to watch for nulls in either of the selected columns. Consider changing the nulls to 0 or 1
before performing the multiplication operation.
Preserving the Top N/Bottom N Rows Based on a Column

To preserve only the Top N / Bottom N rows based on a column (i.e. Total Compensation):
1. Select the column to preserve by ranking.
2. Click the Sort/Filter button for the selected column and click Sort Ascending or Sort Descending.
3. Select Home (tab) → Reduce Rows (group) → Keep Rows → Keep Top Rows / Keep Bottom
Rows.
4. In the Keep Top Rows / Keep Bottom Rows dialog box, enter the desired number of rows to
keep in the Number of Rows field and click OK.

WORKING WITH FILTER – AND & OR CONDITIONS
Filtering by Specific Items

Select the Sort/Filter button at the top-right of a column.
Using the checkboxes, select or deselect the entries according to your need to retain or exclude the
items.
Filtering by Text That Exists

If you need to filter based on the presence of text (ex: all rows that contain the word “Manager”), select
the Sort/Filter button for the desired column and click Text Filters then the applicable filter logic.

Now you can enter the parameters for the text to be discovered, such as “all rows that contain the word
‘Manager’”.
Notice in the above image we are searching for any row that contains the lowercase and proper-case
versions of the word “Manager”. If your text casing had been standardized before this step, you could
list only a single argument for “Manager” in the corresponding casing style.
Creating a Compound Field Filter

Suppose you need to filter against multiple criteria that apply to differing columns in the table (ex: all
Managers with a salary greater than 50,000.)
You could perform two separate filter operations, or you could use the advanced features of the Filter
Rows option.
1. Begin by creating a simple filter based on a single-column criterion (ex: Positions that contain the
word “Manager”.)
2. While in the Filter Rows dialog box, select Advanced to display the Advanced Filter options.
3. On the second criteria row, select the applicable column, operator, and value.
4. Click if you require additional levels of logic.

5. Ensure the And/Or options for each additional level correspond to your desired logic needs.

Beware of Logic Errors

When working with compound AND/OR logic in the dialog box, be aware that Power Query has a very
specific way it likes to deal with logic conflicts.
Example – suppose we want to filter Department to show all rows that are Management or Marketing
and do not contain the word Assistant in the title.
We could set the dialog box as follows:
The logic fails our needs because Power Query applies the AND operator for Position to the
“Department = Marketing” requirement, not the “Department = Management” requirement.
To get Power Query to apply the Position requirement to both Department requirements, we need to
perform two separate filter operations; one for both departments, then another for the Position.
NOTE: If you are feeling adventurous, you can modify the M-Code in the Formula Bar to logically group
the two Department requirements as a single thought, then apply the Position requirement. This is
accomplished by placing a set of parentheses around the two Department requirements.
Table.SelectRows(#"Changed Type", each [Department] = "Management" or [Department] =

"Marketing" and not Text.Contains([Position], "Assistant"))
…becomes…
Table.SelectRows(#"Changed Type", each ([Department] = "Management" or [Department] =

"Marketing") and not Text.Contains([Position], "Assistant"))
Look closely for the opening and closing parentheses in the second version of the instruction.

FILTER SELECTION TRAP

You need to create a list of all Employee names that have the word “Assistant” in the Position.
You do the following:

1. Upload the table to the Power Query editor.
2. Click the Sort/Filter button for Position.
3. Search for “Assistant”
4. Click OK to accept all selected “Assistant” results.

Your end report is what you wanted. You get the list of employees that have the word “Assistant” in the
Position column.
Predicting the Future

There is a problem though! The problem does not affect the current results. The problem lies in the
“future-proofing” aspect of the query. It’s when we add a new position that didn’t exist before that has
the word “Assistant” in it. For example: “Finance Assistant”.

If we never add a new Position with the word “Assistant” for any position other than what we currently
have, we will be okay. This is because of the way Power Query wrote the filter step.
= Table.SelectRows(#"Changed Type", each ([Position] = "Assistant to MD" or [Position] =

"Marketing Assistant" or [Position] = "Procurement Assistant" or [Position] = "Sales Assistant"))
The query is NOT looking for Positions that contain the word “Assistant”, it is instead looking for four
specific Positions that contain the word “Assistant”.
Instead of checkmark selecting the four Positions based on the search results, we should have created a
different search, something like “Text that Contains… ‘Assistant’ ”.
= Table.SelectRows(#"Changed Type", each Text.Contains([Position], "Assistant"))
This will account for future entries that have the word “Assistant” but may also contain words that were
not in the original list.

CHANGE TYPE TRAP – (& REMOVE COLUMNS TRAP)

Power Query will often automatically apply transformations to data that it believes are wanted by the
user.
In most cases, these transformations are welcomed and great time-savers. This is not always the case.
Certain automatic transformations, such as “Change Type”, if done prematurely can cause complications
with later steps.
Change Type Trap – Problem

When we load a dataset into Power Query, an automatic Changed Type step is applied to the columns
to define proper data types.
= Table.TransformColumnTypes(Source,{{"Employee", type text}, {"Department", type text},

{"Position", type text}, {"Salary", Int64.Type}})
The Changed Type step is defining a data type for each column. Nothing wrong with that… until later.
We apply some additional transformations:
• Remove duplicate rows based on Department and Position

• Remove unneeded columns (keep Department and Position)
• Sort results by Department
After the query has processed the data and placed the results in a table, the user renames the Employee
column heading to Name.
When we refresh the query, it fails.

This is because the Changed Type step is attempting to set the data type for a column named Employee,
which doesn’t exist any longer.
Change Type Trap – Solution

Since the Employee column is not retained in the final output, it would be best to not apply data types
to the columns until AFTER the unneeded columns have been removed.
The “future-proof” solution requires us to:
• remove the automatically applied Changed Type step

• remove duplicate rows based on Department and Position
• remove unneeded columns (keep Department and Position)
• sort results by Department
• manually apply data typing to the remaining columns (Transform (tab) → Any Column (group)
→ Detect Data Type.)
Adjusting Automatic Change Type in Power Query Settings

If you do not wish to have Power Query perform the automatic data typing step when load new data or
promoting header rows, you can deactivate the setting at the workbook level or the program level.
Workbook -level Deactivation
1. Select File → Options & Settings → Query Options.
2. In the Query Options dialog box, under the Current Workbook category – Data Load, deselect
“Detect column types and headers for unstructured sources”.

Program-level Deactivation
1. Select File → Options & Settings → Query Options.
2. In the Query Options dialog box, under the Global category – Data Load – Type Detection, select
the option that best suits your needs for most projects.
Remove Columns Trap – Problem

In the previous problem, we have a step where we select the columns we wish to keep (Department and
Position) and we remove all other columns. The “M” code looks like the following:
= Table.SelectColumns(Source,{"Department", "Position"})
Suppose the removal step was constructed with opposing logic. In other words, instead of telling the
query what columns to keep, we tell the query what columns to remove.
= Table.RemoveColumns(Source,{"Employee", "Salary"})
We run the same risk as before where a user renames a column in the source data that no longer
matches the column name in the query step.
Remove Columns Trap – Solution

Since the Employee and Salary columns are not retained in the final output, it’s more robust to define
the columns to retain as opposed to the columns to remove.
This has the added benefit of when new columns are introduced to the dataset, they are also removed
from the final output.

Part 4
Powerful
Power Query
Transformations
Power Query Powerful Power Query Transformations
COLUMN FROM EXAMPLE – TO EXTRACT PATTERNS QUICKLY

The Columns From Example feature is located on the Add Column (tab) → General (group) → Column
From Example.
The purpose of Columns From Example is to use an example (or series of examples) to restructure the
data into a new arrangement based on an established pattern.
When launched, Power Query provides an empty test column by which examples are created.
For our example, suppose we need to extract the first 6 characters from the last name and the first
character from the first name. These need to be concatenated and cased in a Proper Case style (first
letter capitalized).
Although not a requirement, users typically start at the top of the test column and begin creating “the
version they need” via manually typing.

After the user hits ENTER, if the pattern is obvious enough, a faded suggestion list is presented.
If this is the desired output, click or press CTRL-Enter.

If the pattern is more complex, as our example, and Power Query is not offering additional suggestions,
select another cell in the test column and create another example.
It may take 1-5 attempts for Power Query to understand the desired structure, but eventually, it will
succeed (in most cases.)
The feature isn’t perfect; certain patterns elude the feature’s ability to deduce. It is, however, a great
“first attempt” when restructuring data based on a pattern.

Formulas For Free

Once you accept a faded suggestion list from the Column From Example feature, a formula to perform
the data restructuring is created and added to the query.
As you can see, some of these formulas can become quite complicated.
This gives us an appreciation of what we are NOT having to develop when writing queries.
Solving the Problem with the Traditional Approach

If we did not have the Columns From Example feature, and we used the standard Power Query tools,
the following is one way to solve the problem.
1. Split the last name from the rest of the text using Home (tab) → Transform (group) → Split
Column → By Delimiter. Delimiter is a “space” character and the Split At option is from the
right-most delimiter (last space).
2. Capture the first six characters from the last names (Step 1 results) by selecting Transform (tab)
→ Text Column (group) → Extract → First Characters and set the option to 6.
3. Apply the Proper Case format (the first letter capitalized, remaining letters lowercase) by
selecting Transform (tab) → Text Column (group) → Format → Capitalize Each Word.

4. Capture the first character of the first name by selecting the original data column and clicking
Transform (tab) → Text Column (group) → Extract → First Characters and set the option to 1.
5. Merge the results by selecting the column of six characters of the last name, hold CTRL, and
select the first initial column. Next, select Transform (tab) → Text Column (group) → Merge
Columns. There will be no separator used and the new column name will be “LoginID”.
Both methods achieve the same result. Pick the method you are most comfortable with and which you
have the most faith in succeeding.

CONDITIONAL COLUMNS – IN POWER QUERY
If… Then… Else Statements in Power Query

We are presented with the dataset below.
Our objective is:
• Please group the employees under the following criteria.

o LOW = anyone earning below 30K
o AVERAGE = anyone earning between 30K and 50K
o MODERATE = anyone earning between 50K and 90K
o HIGH = anyone earning over 90K
• The results should appear as a new column in the results table.
We can solve this problem by using Conditional Columns.
A Conditional Column is essential an IF statement, where you have a question, an action if TRUE, and an
action if FALSE.
To create a Conditional Column, select Add Column (tab) → General (group) → Conditional Column.
In the Add Conditional Column dialog box, give the result a name (such as “Group”), and define each
level in the logic hierarchy.

Since each condition is looking at the same column (“Salary”) and using the same logic (“is greater than
or equal to”), we only need to define the Values and the Output.
NOTE: Define the logic from the highest value to the lowest; in other words, the most restrictive to the
least restrictive.
The Else field will act as a “catch-all” for any value that does not meet any of the above criteria.

AGGREGATING (GROUPING) DATA – ON MULTIPLE LEVELS

Our objective is:
• Determine how many employees we have in each Department

• Determine how many employees we have in each Group.
• Determine the average salary of each group within the departments.
Although we could answer these questions with a Pivot Table, we can also answer these questions using
Power Query and have the results be delivered as a table.
Launching the Group By Feature

In Power Query, you can launch the Group By feature in several ways:
• Home (tab) → Transform (group) → Group By

• Transform (tab) → Table (group) → Group By
• Right-click a column heading and select Group By…
Using the Group By Feature
NOTE: Be aware that the Group By feature REPLACES your original data with the grouped version of the
data. If you need to retain the original data source, consider making a copy of the original query and
performing the Group By on the copy.
If you select a single column, you will be presented with the Basic view of the Group By feature.
If you select multiple columns, you will be presented with the Advanced view of the Group By feature.

Group By – Basic View

In the Basic view, we need to provide the following information:
• The column to be grouped (ex: Department)

• The heading name of the new column that will hold the grouping calculation (ex: Totals)
• The mathematical operation to be performed on the aggregation column (ex: Sum)
• The column to be aggregated (ex: Salary).
For our example, the results will appear as follows.

Group By – Advanced View

In the Advanced view, we need to provide the following information:
• The columns to be grouped (ex: Department and Group)

• The heading names of the new columns that will hold the grouping calculations (ex: Count and
Average Salary)
• The mathematical operations to be performed on the aggregation columns (ex: Count Rows and
Average)
• The column(s) to be aggregated (ex: Salary).
• For each sub-group you need to create, click the Add Grouping button.
• For each additional aggregation, click the Add Aggregation button.

The results will appear as follows.
Clean up the report with the following transformations:
• Apply a Currency data type to the Average Salary column.

• Create a primary sort by Group.
• Create a secondary sort by Department

GROUP BY FOR ALL ROWS

Our objective is:
• Calculate the salary deviation for each person using the average salary of their department as a
basis
Determining the Average Salary per Department

Select Transform (tab) → Table (group) → Group By and define the following parameters.
If we click OK, all the data will be replaced with the aggregated results.

Creating Nested Tables

Switch to the Advanced view and create another aggregation that uses the All Rows option with the
following parameters.
The result is a table that has a column of nested tables.
Each of the “Table” hyperlinks is a complete table of information, all stored within a single cell. Imagine
every record that pertains to the Department – Sales has been “zipped” into a single object called
“Table” and stored in a single cell.
If you click a “Table” hyperlink, you will extract all the related records into a new table that replaces the
current table.

If you click to the side of a “Table” hyperlink, you can see a preview of the contents of the nested table.
You can extract all the records for each nested table by clicking the Expand Tables button in the heading
of the column of nested tables.
From here, you can select which columns you wish to extract and attach to the existing table.

Since we already possess Department and we don’t require Position, we will select Employee and
Salary. We will not use the column prefixing feature at this time.
The original column information will be repeated for each expanded row.
Calculating the Salary Deviation

To calculate the salary deviation for each employee against their respective department,
1. select the Salary column, press CTRL, and select the Average Salary column.
2. Select Add Column (tab) → From Number (group) → Standard → Subtract.
Final Touches
• Rename the Subtraction column to Deviation.
• Remove the Salary and Average Salary columns.
• Sort the Deviation column from Largest to Smallest (Descending)
• Send the report to Excel as a Table.

UNPIVOT COLUMNS – BASICS

Unpivoting is performed to turn a cross-tabular report into a tabular report.
In simpler terms, we are taking a report that has a category(s) for rows, a different category(s) for
columns, and calculations at the intersections and turning it back into a traditional table format.
NOTE: Unpivoting is NOT the same as Transposing. When you Transpose, you “flip” the categories
between their row/column positions but you are still left with a cross-tabular structure.
Unpivot Example
We have the following dataset:

Highlight the dataset and bring it into Power Query.

The key is to determine which column is to act as our “anchor”; the column by which all remaining
columns are paired against.
For us, the “Product / Sales” column should be repeated for each Month column.
There are two ways we can perform the unpivot operation:
• Select the “anchor” column and click Transform (tab) → Any Column (group) → Unpivot Other
Columns.
• Select all columns except the “anchor” column and click Transform (tab) → Any Column (group)
→ Unpivot Columns.
NOTE: These features can also be accessed by right-clicking the heading of the selected column(s).
In this case, you will get the same results with either method.
To unpivot this into a tabular structure, in Power Query, select the “Product / Sales” column and click
Transform (tab) → Any Column (group) → Unpivot Other Columns.

Trimming the Unpivot Fat

If we scroll to the bottom of our results, we see that the total row was repeated for each month.
There are two ways to deal with these extra, unwanted rows:
• In the results, filter the “Product / Sales” column to not include “Grand Total”.
• Before the unpivot step, filter the “Product / Sales” column to not include “Grand Total”.

UNPIVOT COLUMNS – HOW TO OVERCOME COMMON ERRORS
Common Error #1: Adding Data Post-Data Typing

Examine the following data.
If we perform a typical unpivot operation on the table, we get the results as expected.
The problem occurs when we bring in another month (ex. July)to the data.
When we refresh the query, everything looks correct. Looking at a deeper level, we have a problem.

Examining the Unpivot Query

If we examine the unpivot query, we see at the last step in the query, the data type for Value is no
longer a Decimal data type.
This is because the step that defined the data types were performed before the unpivot step. The
Changed Type step was hard-coded to look for months between “January” and “June”.
Since “July” did not exist when this step was defined, it will not be accounted for in a Refresh operation.

Correcting the Unpivot Logic Error

The better, “future-proofed” version of the query would be to not perform data typing until after the
unpivot step has been performed.
Common Error #2: Dealing with NULLs in the Source Data

Examine the following data and observe the empty cells.
When we bring this table into Power Query, the empty cells are perceived as NULL cells.

In an unpivot operation, cells containing NULLs are omitted from the unpivot result.
But is you require all data to be present after the unpivot step, you need to replace the NULLs with 0
(zeroes).
Before the Unpivot step, select all the columns that could contain NULLs and click Transform (tab) →
Any Column (group) → Replace Values.
In the Replace Values dialog box, the Value To Find will be null and the Replace With will be 0.
The result will be contiguous data, even across empty cells.

Potential Problem for Replacing Nulls with Zeroes

You need to be ever vigilant for query steps that are “hard-coded” to specific column names. This
reduces flexibility when data changes.
We can see that the previously applied Replace Values step is applied only to columns with certain
month names.
When the next month’s data is received (ex. July), the Replace Values step will not be applied to the new
month’s data.
A Little Bit of “M” Goes a Long Way
It’s times like this that knowing just a small amount of “M” code will reap great benefits.
We need to make dynamic the portion of the Replace Values step that looks for months.
Update the formula with the following modification:

Remove all the hard-coded months (including the curly braces) and replace them with the following
function…
Table.ColumnNames(#”Filtered Rows”)

NOTE; In this example, the reference to “#Filtered Row” is the name of the previous step. Replace this
reference with the name of the previous step in your query.

PIVOT COLUMN – CREATE READABLE REPORTS FAST

When you pivot a column, you create additional columns based on the unique values of an original
source column.
A benefit of this process is the ability to inject data aggregation.
We begin with a tabular structure.
The most important step is to determine which column to pivot. In other words, which column in the
table should become the new columns and headings in the cross-tabular report.
For our data, we want the Products to be listed down the left side of the report while the Months are to
be listed across the top of the report. The Sales will become the intersecting aggregations.
We need to select the Months before the pivot operation. Selecting Months informs Power Query to
create a unique list of items from the selected columns and create a new column for each item.

Creating a Pivot Report

Below is our dataset.
Click in the data and load it into Power Query.

Select the column to be pivoted (ex: Month) and click Transform (tab) → Any Column (group) → Pivot
Column.
In the Pivot Column dialog box, select the field that will serve as the aggregations for the row/column
intersections.
You can also select Advanced Options to get the ability to select the aggregation method (i.e. SUM,
AVERAGE, COUNT, etc.)

The result is a cross-tabular report that is easier to understand than a flat table.

SPLIT COLUMN – THE PROBLEM THAT’S EASY TO MISS

We are presented with the following dataset.
Our objective is to count the number of Available Colors for each Product.
The Problem
The problem is that the report designer has inserted in-cell carriage returns (ALT-Enter) in the cells that
hold colors.
The Solution
In Power Query, perform the following steps:
1. Select the Available Colors column.
2. Select Home(tab) → Transform (group) → Split Column → Split Column by Delimiter.

3. In the Split Column by Delimiter dialog box, observe that Power Query has determined the
process needed to split each color by the embedded in-cell carriage return characters.
4. Click OK.
The result is a cross-tabular report that separates each product’s colors into separate columns.
Next, we will revert this cross-tabular report into a tabular report (table).
5. Right-click the header for Product and select Unpivot Other Columns.
6. Remove the unneeded Attribute column.

Next, use the Group By feature to count the number of colors for each product.
7. Select Transform (tab) → Table (group) → Group By.
8. In the Group By dialog box, select Product as the column to be grouped and Count Rows as the
operation to be performed for each Product.

The Flaw In the Logic

Although the query appears to be working, there is a potentially disastrous flaw in the logic.
If we introduce a new color to the dataset that was not present during the construction of the query, the
new color is not reflected in the results.
The fault lies in the Split Column by Delimiter step.
The columns to create for storing colors are restricted to three columns (the maximum number of colors
for any given product.)
This needs to be made dynamic so it can account for any number of supplied colors.
The Fix for the Flaw

To account for a variable number of colors, we need to learn a little-know trick feature of Split Column
by Delimiter. That’s in the following section.

SPLIT COLUMN BY DELIMITER – ROWS INSTEAD OF COLUMNS

This section will solve the problem presented in the previous section where we needed to account for a
variable number of colors assigned to a given product.
The colors are listed in a single cell separated by an in-cell carriage return (ALT-Enter).
Splitting the Colors Across Rows

Instead of creating a new column for each product color, we will create a new row for each product
color.
Starting with the original dataset, in Power Query, perform the following steps:
1. Select the Available Colors column.
2. Select Home(tab) → Transform (group) → Split Column → Split Column by Delimiter.
3. In the Split Column by Delimiter dialog box, observe that Power Query has determined the
process needed to split each color by the embedded in-cell carriage return characters.

4. In the Advanced Options section, change the Split Into option from Columns to Rows.
5. Click OK.
The result is as follows.
An advantage of this technique is that it saves the steps required to perform the unpivot operation.
A disadvantage is the presence of NULL values in the Available Colors column.

Tweaking the List

To ensure we do not include the products without colors, filter the Available Colors column to not
include NULL values.
Next, we will perform the same steps we performed in the previous section to create the grouped list of
products and their color counts.

“Future-Proofing” the Query

Suppose an existing color is added to a product in the dataset where that color already exists. The query
will not treat the duplication of colors a redundant and ignore the repeated color. The query will treat
every color as unique.
To account for this possibility, we need to perform a Remove Duplicates operation on the color before
the grouping step.
To make this even better, we will standardize the casing in the event users do not follow the same casing
patterns.
6. Select the Split Column by Delimiter step.
7. Select Transform (tab) → Text Column (group) → Format → Capitalize Each Word.
8. Select .
9. Select both the Product column and Available Colors column.
10. Select Home (tab) → Reduce Rows (group) → Remove Rows → Remove Duplicates.
11. Select .
12. Select the last step in the query (Grouped Rows).

The results are now “future-proofed”.

Part 5
Date and Time

Transformations
Power Query Date and Time Transformations
DATE AND TIME TRANSFORMATIONS

Power Query possesses a fantastic library of date and time transformation features. Although Excel can
replicate these tasks with an array of simple to highly complex formulas, Power Query can produce
these results, in most cases, with a single click of the mouse.
There are three major categories of transformations:
Date Transformations Time Transformations Duration Transformations
You can use these from the Add Column tab or Transform tab depending on whether you’d like to add a
new column for the transformation or update the values in the existing column.

Making the Complex Simple

Below are a few examples demonstrating where Power Query can perform calculations with only a click
that would otherwise prove quite difficult for users working with traditional Excel formulas.
Calculating Week of Month
Suppose we want to calculate the week number that a week falls within its month (ex: May 6, 2020
would occur in the 2nd week of May, whereas May 27, 2020 would occur in the 5th week of May.)
Selecting a column of dates and clicking Add Column/Transform (tab) → Date & Time (group) → Date -
→ Week → Week of Month gives us exactly what we need without all the stress of writing a
complicated formula.
The Excel formula equivalent to this one-click action would require writing the following formula:
Calculating Hours Worked

Another “complicated-made-easy” task in Power Query is to determine the number of hours between
two sets of date/time entries. This can be accomplished in two steps.
Step 1: Determine the difference between the dates by selecting the two date/time columns and click
Date → Subtract Days.

Step 2: Convert the results to display as the difference in hours by selecting the results of Step 1 and
click Duration → Hours.

USEFUL DATE FEATURES
Feature Use
Age Return the duration between the current local time and the values in the
selected columns
Date Only Extract the Date component from the Date/Time values
Parse Return a Date value parsed from the text
Year Extract the Year component from the Date/Time values
Start of Year Return the first day if the year corresponding to each Date/Time value
End of Year Return the last day if the year corresponding to each Date/Time value
Month Extract the Month component from the Date/Time values
Start of Month Return the first day if the month corresponding to each Date/Time value
End of Month Return the last day if the month corresponding to each Date/Time value
Days in Month Return the number of days in the month corresponding to each Date/Time
value
Name of Month Return the name of the month corresponding to each Date/Time value
Quarter of Year Return the quarter corresponding to each Date/Time value
Start of Quarter Return the start of the quarter corresponding to each Date/Time value
End of Quarter Return the end of the quarter corresponding to each Date/Time value
Week of Year Return the week of the year corresponding to each Date/Time value
Week of Month Return the week of the month corresponding to each Date/Time value
Start of Week Return the start of the week corresponding to each Date/Time value
End of Week Return the end of the week corresponding to each Date/Time value
Day Extract the Day component from the Date/Time values
Day of Week Return the day of the week corresponding to each Date/Time value
Day of Year Return the day of the year corresponding to each Date/Time value
Start of Day Return the start of the day corresponding to each Date/Time value
End of Day Return the end of the day corresponding to each Date/Time value
Name of Day Return the name of the day corresponding to each Date/Time value
Subtract Days Return the number of days between the values in the first and second
selected columns
Combine Date and Time Merge the selected columns into a new column containing both Date and
Time data
Earliest Return the earliest Date value
Latest Return the latest Date value

Date Calculation Examples

Example 1: Determine Month Name of Date
Select a column of dates and click Date → Month → Name of Month.
Example 2: Calculate the Quarter a Month Occurs

Select a column of dates and click Date → Quarter → Quarter of Year.
Example 3: Determine Weekday Name of Date

Select a column of dates and click Date → Day → Name of Day.

USEFUL TIME FEATURES
Feature Use
Time Only Extract the Time component from the Date/Time values
Local Time Return the Date/Time/Timezone values
Parse Return the Time value parsed from the text
Hour Extract the Hour component from the Date/Time values
Start of Hour Return the start of the hour corresponding to each Time value
End of Hour Return the end of the hour corresponding to each Time value
Minute Extract the Minute component from the Date/Time values
Second Extract the Second component from the Date/Time values
Subtract Contains the duration between the values in the first and second selected
columns
Combine Date and Time Merge the selected columns into a new column containing both Date and
Time data
Earliest Return the earliest Time value
Latest Return the latest Time value

Time Calculation Examples

Example 1: Determine the Earliest Time
Select the columns of times and click Time → Earliest.
Example 2: Calculate the Start of an Hour

Select a column of dates and click Time → Hour → Start of Hour.
Example 3: Combine Date and Time Information Into a Single Column

Select a date column and a time column and click Date → Combine Date and Time.

USEFUL DURATION FEATURES
Feature Use
Days Return the days component corresponding to each Duration value
Hours Return the hours component corresponding to each Duration value
Minutes Return the minutes component corresponding to each Duration value
Seconds Return the seconds component corresponding to each Duration value
Total Years Return the total number of years in each Duration value
Total Days Return the total number of days in each Duration value
Total Hours Return the total number of hours in each Duration value
Total Minutes Return the total number of minutes in each Duration value
Total Seconds Return the total number of seconds in each Duration value
Subtract Contains the duration between the values in the first and second selected
columns
Multiple Multiplies each value in the selected columns by a specified value
Divide Divides each value in the selected columns by a specified value
Statistics (Sum, Min, Sum, Min, Max, Median, or Average of all the Durations
Max, Median, Average)
Durations are represented in the following format:
Days.Hours:Minutes:Seconds
Because of this, it is common to reformat the duration to something more easily understood, such as
Total Years, Total Days, or Total Hours.

Duration Calculation Examples

Example 1: Reduce All Time Worked by 50%
Select a column of durations and click Duration → Multiple → .5 (or Divide → 2).
Example 2: Calculate Average Time

Select the columns of durations and click Duration → Statistics → Average.
Example 3: Determine the Age of Employees

Select a date column and click Date → Age. Next, select the Age column and click Duration → Total
Years. Finally, select the Total Years column and click Rounding → Round Down.

Part 6
Custom
Columns
Power Query Custom Columns
WHY USE CUSTOM COLUMNS
Your Imagination has No Limits

When using the features available from the Add Column tab/ribbon, the data being manipulated
remains unchanged while the results appear in a new column.
We can produce new columns of information in a variety of ways.
• Use a Conditional Column to produce data based on a test (like an IF statement)

• Create columns from examples, like the Flash Fill tool in Excel
• Extract the Year from a Date
• Create a list of numbers using the Index Column feature
These are wonderful features, but there will be times when the column you need to create is beyond the
scope of the tab/ribbon tools. This is when you need to create the custom column using a formula.
The Custom Column dialog box is accessed by selecting Add Column (tab) → General (group) →
Custom Column.
Within the Custom Column dialog box, you can create unlimited functions to produce virtually any
output needed.
= ( [Cost of Living] + [Weather] + Number.From([Health Care Quality]) + [Crime] ) / 4

For a complete list of available Power Query functions along with detailed explanations and examples,
visit the following Web resource.
docs.microsoft.com/en-us/powerquery-m/power-query-m-function-reference
The library can also be accessed by clicking the link in the lower-left corner of the Custom Column dialog
box labeled “Learn about Power Query formulas”.
NOTE: Custom Columns can also be used to create more complex query steps that combine several
click-based steps into a single step. This can result in reduced code and increased performance. It does,
however, require a more advanced understanding of the “M” language.

CUSTOM COLUMNS – TYPE COMPATIBILITY & DATE FUNCTION
Data Type Compatibility

When concatenating data from different columns, it is important to ensure that the data types match.
Power Query does not allow concatenating dissimilar column types, such as Text to Numbers, or Dates
to Boolean.
Unlike Excel, Power Query is type-sensitive as well as case-sensitive.
Data Type Conversions using Functions

Power Query contains functions that will convert data from one type to another. We can use these
functions to standardize our data before concatenation.
Numbers to Text
To convert a number to text, we can use the Text.From function.
Text.From( [Emp ID] )
This could be used in a larger context, such as…
=[Last Name] & Text.From( [Emp ID] )

Text to Numbers
To convert text to a number, we can use the Number.From function.
Number.From( [TaxRate] )
This could be used in a larger context, such as…
=Sales * Number.From( [TaxRate] )
Date Function
In a dataset where we have Year and Month as separate columns, formatted with a Whole Number
data type, we wish to create a date.
Since we are missing DAY information, we will fix the days to the 1st for each date.
The function and syntax for this date creation are as follows.
= #date( [Year], [Month], [Day])
Our example will use the Start Year and Start Month columns while using a fixed value of 1 for the day.
= #date( [Start Year], [Stary Month], 1)

Other Intrinsic Functions

Below are other intrinsic functions worth investigating from the Power Query Function Library:
• #duration
• #datetimezone
• #date
• #time
• #dateTime
For a complete list of available Power Query functions along with detailed explanations and examples,
visit the following Web resource.
docs.microsoft.com/en-us/powerquery-m/power-query-m-function-reference
The library can also be accessed by clicking the link in the lower-left corner of the Custom Column dialog
box labeled “Learn about Power Query formulas”.

Part 7
Power Query
Online Data
Sources
Power Query Power Query Online Data Sources
CONNECTING TO DIFFERENT SOURCES
The Vast World of Connectors

One of Power Query’s greatest features is the ability to directly connect to a variety of different data
sources. For example:
• Web pages
• Embedded files within a Web page
• SharePoint
• OneDrive for Business
• Outlook email
• OData feeds
The list of connectors is vast and ever-increasing.
Keep in mind is that not every connector is available to all versions of Power Query. Certain connectors,
like “connect to a Web page” are universally available, while other connectors like “SharePoint Online
List” are limited to later or higher-level versions of Office.
The below link will provide a list (current as of Spring 2020) of connectors and their platform availability.
https://support.microsoft.com/en-us/office/where-is-get-transform-power-query-
e9332067-8e49-46fc-97ff-f2e1bfa0cb16

IMPORT DATA FROM A WEBSITE

More and more data are being hosted online and keeping local copies of online data current can be a
never-ending challenge.
The Web connector allows us to use the most current version of Web data without worrying about local
copy synchronization.
Connecting to a Website
In most cases, the URL used to view the data in a Web browser is the same URL needed for Power Query
to retrieve the data.
We launch the connector by selecting Data (tab) → Get & Transform Data (group) → From Web.
In the From Web dialog box, enter the URL and click OK.
Website Access / Security

When connecting to a website for the first time, you will need to define the level of security and possibly
the access credentials (i.e. name and password).
NOTE: Power Query requires that tables on a website be formatted using HTML to be detected and
imported.
Connecting to an Embedded Excel File or Text File on a Webpage

If the website data is only available via a file download, the data connection to the website is lost once
the file has been downloaded. The file would need to be re-downloaded every time updates are
desired.
We can solve this problem by connecting directly to the file’s download address.
To obtain a file’s download address, right-click on the file’s hyperlink and select “Copy Link” (your
browser’s label for this option may be different.)
Once the link is obtained, you can use that captured link in the URL field of the From Web dialog box.
Updating Connection Credentials

In the event you need to update your connection credentials (password changes, use a different
username, etc.), from Excel select Data (tab) → Get & Transform (group) → Get Data → Data Source
Settings.
In the Data Source Settings dialog box, select the desired connection from the Data Sources list and click
“Edit Permissions…”

IMPORT DATA FROM ODATA (OPEN DATA PROTOCOL)

The Open Data Protocol allows you to access data models that are hosted on the Web.
The link for ODATA is…
https://www.odata.org
Several sample data models are available for testing purposes. These are similar to data models that
would require complex SQL servers working locally to host and allow access.
The main website provides access to these test data modes by selecting Developers → Reference
Services.
Connecting to ODATA
To connect to an ODATA feed, select Data (tab) → Get & Transform (group) → Get Data → From Other
Sources → From ODATA Feed.
NOTE: The data models available on the ODATA website can be accessed anonymously.

GET GOOGLE SHEET DATA WITH POWER QUERY
Obtaining the Google Sheets URL

In Google Sheets, select File (tab) → Publish to the Web.
In the Publish to the Web dialog box, we can elect to publish the document in a variety of formats; web
page, .CSV, .text, PDF, Excel, and .ODS.
Once the format is selected, click Publish to obtain the URL that grants access to the data.
Copy this link and use it in the following step.
Connecting to the Google Sheets URL

In most cases, the URL used to view the data in a Web browser is the same URL needed for Power Query
to retrieve the data.
We launch the connector by selecting Data (tab) → Get & Transform Data (group) → From Web.
In the From Web dialog box, enter the Google Sheets URL and click OK.
Because Google Sheets have a similar structure to Excel workbooks, it is necessary to select a sheet (or
sheets) of which to import.
NOTE: When making changes to the source material in Google Sheets, it may take upwards of 5 minutes
for the changes to become available to the query when refreshing the data model in Excel.

CONNECT TO MICROSOFT EXCHANGE – GET OUTLOOK EMAIL
Extracting Content from Email Messages

Connecting to Exchange allow for some interesting process automation.
• Analyze survey results that are returned via email distributions.

• Automate attachment retrievals
NOTE: The following process requires a valid Microsoft Exchange account.
To access the Inbox, folders, and emails from your Exchange account, select Data (tab) → Get &
Transform (tab) → Get Data → From Online Services → From Microsoft Exchange Online.
If you are running a local Outlook installation, you can select Data (tab) → Get & Transform (tab) → Get
Data → From Other Sources → From Microsoft Exchange.
In the Microsoft Exchange dialog box, enter your email address and click OK.
At this point, you will need to provide authentication credentials. These can be your local Active
Directory / Exchange credentials or your Microsoft account credentials.

Once connected, we see the major functional areas of the Exchange account. We can click Mail (on the
left) to display the folders and items in the Exchange Inbox. Notice the Headings of the Inbox preview.
Once in Power Query, we can search for items based on…
• Folder location (ideal when using Outlook Rules)

• date of receipt
• name of sender
• contains attachments
• keywords in subject or body text
• importance/priority

CONNECT TO SHAREPOINT OR ONEDRIVE FOR BUSINESS
Extracting Content from Cloud-based Storage

Connecting to SharePoint Has the following advantages:
• Easily share files with coworkers where the file’s size would be too large for email attachments.
• Allow for simultaneous, shared access to files.
NOTE: The following process requires a valid Microsoft Business account with access to SharePoint or
OneDrive for Business.
The most difficult part of this process is locating the path for use in Power Query. One common
technique is as follows.
1. Open a browser and access your SharePoint drive or OneDrive for Business drive.
2. Locate the file by which you require a connection.
3. Click the “three stacked dots” to the right of the file’s name.
4. In the menu, select Details.

5. In the details panel (right), scroll to the bottom to locate the Path.

6. Click the icon to the right of Path to copy the link address to memory.
To access the SharePoint file, select Data (tab) → Get & Transform (tab) → Get Data → From Web and
paste the copied link into the URL field and click OK.
NOTE: Because the source pointer is not pointing to your local computer, if you share this file and a user
attempts a refresh of the data, they will be required to enter their SharePoint or OneDrive for Business
credentials.

Part 8
Combining &
Appending Data
Power Query Combining & Appending Data
WHY APPEND DATA? THE DIFFERENCE BETWEEN APPEND AND MERGE
Appending Data
When you append data, you “stack” the results. It would be like taking multiple identical tables (same
columns/same headings) and place one table below the other. Appending grows tables in rows.
Things to be aware of when appending results from separate queries:
• You must have the same number of columns. If you don’t, rows from one table that do not have
entries for a column in the other table will contain nulls in the non-matching column.
• The column heading’s names must be the same. If the names are different, Power Query will
treat the columns as separate columns.
• The names must match in casing (formatting). Power Query is case-sensitive. Differences in the
casing will result in separate columns of potentially duplicate information.
• The columns should have matching data types. Matching columns with dissimilar data types will
result in a loss of data type definition. Results will be presented as the “Any” data type.
• The columns do NOT need to be in the same order. Columns will be matched based on the
heading names and formatting.
Once you have created and processed the separate datasets, you have two options when appending the
results into a single table:
• Append Queries
• Append Queries as New
Append Queries
This option will attach the results of one query to the bottom of the results of another query. You are
altering one of the queries’ dataset, but no additional queries are being created.
Append Queries as New
The combined results of the original datasets are stored in a new query’s results.

Creating the Appended Query

To create an appended query in Excel (this produces a new query):
Select Data (tab) → Get & Transform (group) → Get Data → Combine Queries → Append.
To create an appended query in Power Query:
1. Select Home (tab) → Combine (group) → Append Queries/Append Queries as New.
2. In the Append dialog box, select the two query results you wish to append to one another and
click OK.
If you need to append more than two query results, select “Three or more tables” and place the desired
queries in the “Tables to append” list (right).

Merging Data
When you merge data, you find matching records in separate tables and combine their content into a
single table record. This is like performing a VLOOKUP (or any other type of lookup operation) in Excel.
Merging grows tables in columns.
Things to be aware of when merging results from separate queries:
• The tables being merged must have at least one column in common (ex: Part Number).
• The common columns must be of the same data type.
• Records of one table with no matching row in the other table will contain nulls in the merged
columns.
Once you have created and processed the separate datasets, you have two options when merging the
results into a single table:
• Merge Queries
• Merge Queries as New
Merge Queries
This option will attach the results of one query to the side of the results of another query. You are
altering one of the queries’ dataset, but no additional queries are being created.
Merge Queries as New
The combined results of the original datasets are stored in a new query’s results.
Creating the Merged Query

To create a merged query in Excel (this produces a new query):
1. Select Data (tab) → Get & Transform (group) → Get Data → Combine Queries → Merge.
To create a merged query in Power Query:
1. Select Home (tab) → Combine (group) → Merge Queries/Merge Queries as New.
2. In the Merge dialog box, select the two query results you wish to merge.
3. Select the matching columns from the top and bottom queries.
4. Select the Join Type (ex: Left Outer).


Part 9
Merge Options
Join Kinds
Explained
Power Query Merge Options – Join Kinds Explained
OVERVIEW OF MERGE OPTIONS – UNDERSTANDING JOIN KINDS

The Merge feature can be accessed in Power Query by selecting Home (tab) → Combine (group) →
Merge Queries.
The Merge feature can also be accessed from within Excel by selecting Data (tab) → Get & Transform
(group) → Get Data → Combine Queries → Merge.
REMEMBER: Merge expands your data left-to-right (columns), where Append expands your data
top-to-bottom (rows).
NOTE: Connecting columns (key fields) must be the same data type.
Advantages of Merges
• Merges can occur between different external data sources
• Merging is faster on larger data sets compared to performing a traditional lookup operation
• The user is not required to write any complicated (or simple) formulas
Merge Options
Power Query is equipped with the following merge strategies:
• Left Outer (all from first, matching from second)

• Right Outer (all from second, matching from first)
• Full Outer (all rows from both)
• Inner (only matching rows)
• Left Anti (rows only in first)
• Right Anti (rows only in second)
Reading the parenthetical explanations usually makes more sense than the official titles.

Left vs. Right / First vs. Second

When working with relational databases, it is a tradition to refer to the tables being merged as “left”
table and “right” table.
In Power Query, likely due to the design of the Merge Queries interface, the tables are referred to as
“first” table and “second” table.

Understanding Join Kinds

Left Outer
Keep all records from the first table and matching records from the second table.
Right Outer
Keep all records from the second table and matching records from the first table.
Full Outer
Keep all records from both tables.
Left Anti
Keep records that only appear in the first table.
Right Anti
Keep records that only appear in the second table.
Inner
Keep records that only appear in both tables.

MERGE BASED ON MULTIPLE COLUMNS
Creating a Compound Key

Merging based on a single column may likely produce issues when multiple matches appear in both the
first and second tables.
To help differentiate the records, perform the merge by matching multiple columns (ex: FirstName &
LastName to matching FirstName & LastName).

This can be accomplished by selecting the first column to match (ex: LastName) in the Merge dialog box,
press and hold the CTRL key, then click the second column to match (ex: FirstName).
You are not limited to two columns for greater specificity; you can select as many columns as needed to
achieve uniqueness.
NOTE: Because the act of expanding the fields of the merged table will likely reorder the rows of the
companion table, you may wish to add an index column before expanding the merged table. This will
allow you to restore the list to its original order. Once restored, you can remove the index column.
MERGE TO GET MULTIPLE MATCH RESULTS
Dealing with One-t-Many Results

There will be times when you have a “left” table with unique entries in the key column merging with a
“right” table that contains repeated uses of the same keys with different associated information.
When we perform a Merge with a Left Outer join, we can see that there are multiple “hits” against
Employee ID “67”.
When you extract the nested table’s content, you will receive all matching rows.

Retrieving the Match with the Latest Date

Suppose you only want the matching record with the latest Start Date?
1. In the list of queries, create a Reference to the TableMaster query and rename it
“TableMasterLatestDate”.
We only want to retain the row with the latest date for any duplicate Employee IDs.
The only columns needed are the Employee ID and Start Date columns.
2. Select the Employee ID and Start Date columns and remove all other columns.
3. Select Transform (tab) → Table (group) → Group By.
4. Set the Group By dialog box parameters as listed in the following image.
This yields a table with 1 record per Employee ID.

5. To retrieve the missing information, merge the results of this table with the original TableMaster
table using Employee ID and Latest Date/Start Date as the keys.
After extracting the needed fields from the nested tables, the results are as follows.
6. Save the results as a Connection Only query.

7. Merge the TableTeam query results with the TableMasterLatestDate query results using a Left
Outer join.

8. Expand the nested tables with the needed fields.

MERGING TEXT COLUMNS – DANGERS & PITFALLS
It’s the Little Things that Get Us

Beware of the following issues that will disrupt successful merge operations:
• Matching items of each list’s key column must be of the same data type. Power Query is
type-sensitive.
• Matching items of each list’s key column must match in casing (i.e. uppercase, lowercase, proper
case). Power Query is case-sensitive.
• Extraneous spaces before or after the data can cause “invisible” differences when matching
items. Consider running a TRIM operation before performing the Merge operation.
• Non-printable characters in the text (i.e. carriage return or line feed characters) before or after
the data can cause “invisible” differences when matching items. Consider running a CLEAN
operation before performing the Merge operation.

HOW TO USE FUZZY MATCH
When Close is Good Enough

As seen in the previous page’s list of “Dangers & Pitfalls”, if the data from one table cannot be located
EXACLTY in the merging table, a match will not take place for that item.
What if we could instruct the Merge feature to think more like a human. In other words, if an entry is
really close, consider it a match and make the connection.
For example, “Walter Miller” and “Walter J. Miller” can be assumed to be the same person. Likewise,
“Alissa Mitchell” and “Alissa mitchell” would be considered a match.
This is where “fuzzy matching” comes into play.
While in the Merge dialog box, the “Fuzzy Match” options are located directly below the Join Kind
dropdown.

Fuzzy Match Options

• Similarity Threshold (optional) – This setting is a decimal number threshold between 0.00 and
1.00. 0.00 will cause all values to match each other, while 1.00 will only permit exact matches.
The default value is 0.80. You may need to experiment with various threshold levels to achieve
your desired level of forgiveness. Beware: setting the threshold too low may result in excessive
false positive matches.
• Ignore Case – Disregard any case differences.
• Match by Combining Text Parts – Attempts various combinations of the supplied text to achieve
a match.
• Maximum Number of Matches (optional) – A setting of 1 will only accept the first encountered
potential match. Higher values will potentially return multiple matches on the same item. This
setting is valuable when setting the Similarity Threshold to a low value.
• Transformation Table (optional) – This allows the use of a concordance table to correlate
matches from a pre-defined list. The list must include two columns; the first column labeled
“From” and the second column labeled “To”. This would allow entries like “MS” and “MSFT” to
force match to “Microsoft”.
Using a Transformation Table

To use a Transformation Table during a Merge operation, a query needs to exist that points to a 2-
column table labeled “From” and “To”.
This can be a loaded table or a Connection Only table.

Select this table from the dropdown as indicated below.

Part 10
When to use
Power Pivot &
Load to Data
Model
Power Query When to use Power Pivot & Load to Data Model
WHEN TO LOAD TO THE DATA MODEL
Out of Sight, but Not Out of Mind

The Data Model is described by Microsoft as:
A Data Model allows you to integrate data from multiple tables, effectively building a
relational data source inside an Excel workbook.
Within Excel, Data Models are used transparently, providing tabular data used in
PivotTables and PivotCharts.
A Data Model is visualized as a collection of tables in a Field List, and most of the time,
you’ll never even know it's there.
The Data Model is used primarily by Power Pivot and Power BI.
Power Pivot and the Data Model

Power Pivot is identical to traditional Pivot Tables but with the following additional features:
• Data can span well beyond the 1 million row limit of an Excel worksheet.
• Pivot tables can be built from elements originating from different tables.
Adding Query Results to the Data Model

Query results can be added to the Data Model by select the option labeled “Add this data to the Data
Model” in the Import Data dialog box.
NOTE: Unless you have a compelling reason to see the table in a worksheet, it’s better to create a
connection to the data and only load it into the Data Model. This will save on memory and storage.

Power Query When to use Power Pivot & Load to Data Model
When to Use the Data Model

Traditionally, all data had to be contained in a single table for features like Pivot Tables or Pivot Charts to
function properly.
If you were collecting data from multiple tables (ex: sales, prices, colors, store name, etc.), you would
need to perform multiple merge operations to collect all the data into a single table.
This is a perfect opportunity to use the Data Model.
By leaving all data in their original tables, we can create relationships between the tables.
Is Power Query Needed to Load Data into the Data Model

The short answer is “No”.
If you have your data stored in Excel Tables and they do not require further transformations, you can
create relationships between them using the Relationships feature on the Data ribbon.
The interface is not as user-friendly, but the basic connectivity tools exist.

Part 11
Understanding
M Formula
Language
Power Query Understanding M Formula Language
M LANGUAGE – HOW M THINKS (LET EXPRESSION & VALUES)
Viewing Code in the Advanced Editor

When working with a query, steps are displayed in the Applied Steps panel as user-friendly (although
sometimes not the most descriptive) labels.
When we select a step from the list, we can see the underlying M code in the Formula Bar.
NOTE: If you do not see the Formula Bar in the Power Query Editor, select View (tab) → Layout (group)
→ Formula Bar.
If you want to see the entire code set at once, instead of clicking step-by-step while you try to remember
what you read from another step, you can activate the Advanced Editor by selecting Home (tab) →
Query (group) → Advanced Editor.

This will display the code in its entirety.
NOTE: Consider experimenting with the Display Options in the Advanced Editor. Features like “Display
Line Numbers” are especially helpful.
Expressions
The let Expression
The purpose of the let expression is to return a value. The returned value can be as small as a single
number or letter, or as large as a table with millions of rows and thousands of columns.
Each line in the code is a separate instruction ending with a comma except for the LAST instruction.

Think of the comma as the word “and”. You could say, “Do this AND do this AND do this.” You wouldn’t
put an “AND” at the end because that implies another step should occur.
Storing the Individual Step Results

Each step must store its results in a named place in memory.
Just like Excel must store the results of a formula in a cell address, Power Query stores the results of a
step in a variable name.
Excel Example:
A1 = B1 + B2
Power Query Example
Source = Excel.CurrentWorkbook(){[Name="TEmployee"]}[Content],
NOTE: If you have but a single step query, you can omit the let, in, and named memory location from the
M code. This is not encouraged, but it’s an interesting behavior.
The in Expression
After the final step in the query, the in expression is used to signify the end of the steps and where to
retrieve the final result. Typically, the final result is the memory name used in the final step.
It’s a way of saying, “Get the last result produced and deliver it to the user.”

Comments in M Code
To add a single-line comment, precede the comment with a double forward slash.
To add a multi-line comment, start with the “/*” characters and end with the “*/” characters.
Step Name Syntax

If a step name includes any spaces, you must enclose the full step name with double-quotes and a
leading “#” character.
#”Changed Type”
If the step name does not contain spaces, these characters can be omitted.
ChangedType
NOTE: Omitting the spaces tends to make the code cleaner, easier to read, and less prone to syntax
errors.
WARNING: If you manually modify the name of a step in the Advanced Editor, remember to modify any
other steps that reference the renamed step. Otherwise, your code will fail.
“M” Characteristics

Lazy Code Execution

If a line of code’s (i.e. a query step) result is not used by other steps in the query, that line of code is not
executed.
Code Variations
When performing manipulations on the date, Power Query will write the code in a way it thinks is most
efficient.
Example:
If we were to filter out a single item, the code is likely to read as “do not include this one item”.
= Table.SelectRows(#"Filled Down", each ([Position] <> "Accountant"))
But if we filtered out most items, the code is likely to read as “only keep this one item”.
= Table.SelectRows(#"Merged Columns", each ([Position] = "Accountant"))
This can have unintended side-effects when adding additional data. You may inadvertently include the
new data when you didn’t want to, or you may omit the new data when you needed it to be part of the
result.
Step Result Retention
Upon code completion, except for the final step result, the individual step results that were stored in
named memory locations are discarded.
This allows for more efficient use of memory by not retaining step results that are no longer needed.

DEFINING & INVOKING CUSTOM M FUNCTIONS

As seen earlier, the let expression returns a value. Functions also return a value.
Function Syntax
The beginning of a function requires the establishment of variables (named places in memory) to accept
input for processing. These must be contained within a set of parentheses, separated by commas, and
followed by “=>” characters.
(Price, TaxRate) =>
After the “=>” characters are the instructions for processing the supplied data.
This produced an interface that allows the user to manually type the variables.
When we supply data into the fields and press , we see the result of the function as a separate
query.

If we supply data that cannot be processed by the function (ex: text where it expects a number), we
encounter an error.
This can be mitigated by requiring specific data types for each variable.
(Price as number, TaxRate as number) =>
NOTE: As with a traditional query, if the function is but a single line, the need for let and in statements is
not required.
The input interface now instructs the user as to the desired data types and alerts the user when invalid
data is supplied.

Multi-Step Functions
If we wish to perform several steps when transforming the data, we must include the let, in, and named
memory address where the final result is stored.
Saving a Function
When executing a Close & Load on a function, the query is always stored as a Connection Only query.
Using a Function in a Query

To apply the steps of a function to another query’s data, select Add Column (tab) → General (group) →
Invoke Custom Function.

Supply the results column a name and define the fields to be used by the related function arguments.
We see a new step has been added to the query that invokes the custom and applies the function logic
to each row in the table.
The invoked custom function code can be altered to incorporate additional actions (ex: data typing).

Updating Function Logic

Be aware that if you alter the logic of the function (i.e. change the tax rate), all queries that have used or
will use the function will obtain a new result.
Embedded Functions Within Queries

Custom Functions are not required to be stand-alone objects; they can be part of a let statement within
a larger query.
NOTE: The function can reside anywhere within the let..in section. The function will not be executed until
it is called by another line of code.

REFERENCE GUIDE FOR STANDARD M FUNCTIONS

Just as Excel has a library of built-in functions, M has a similar library of built-in functions. One
fundamental difference lies in the way the functions are referenced.
In Excel, if you wanted to invoke the YEAR function, you would write something like the following:
=YEAR( A1 )
We are not required to tell Excel which category the YEAR function resides because all functions reside
on the same list.
With M being the “new kid on the block”, Microsoft took advantage of its youth and has created
different categories for different functions. This allows M to reference the same named function (ex:
FROM), but get different results based on the category from which FROM was retrieved.
In other words, the results you get when using the “From” function will be different depending on
whether you used Date.From, Byte.From, Duration.From, or Number.From.
Learn About Power Query Formulas

When working in the Power Query Editor and writing or editing a custom column, you can access the
Microsoft Power Query M Function Reference by clicking the link in the lower-left of the Custom
Column dialog box.

This will take you to Microsoft’s reference library on the Web.
Clicking a function category will present a list of all functions within the selected category.

Clicking a function will present a detailed explanation of the function, the function arguments, as well as
examples of use.
To locate a specific function or category in the library, a Search box is provided in the top-left of the
page.

Built-In Function Documentation

To access the built-in function documentation, start a blank query and type the following in the Formula
Bar:
= #shared
This will present the list of categories and related functions.
This tends to be a bit much to deal with, so we can make things easier by converting the list to a table.
This provides us with a table that we can filter based on numerous criteria.

Clicking to the right of a function’s hyperlink will display the full documentation of that function in the
lower panel of the Power Query Editor.
Function Context
When using a function, be mindful of the function’s context (i.e. category). Functions that begin with
Text, List, Record, or Table will view the data differently.
• Text – applies the function to a specific cell in a table.

• List – applies the function to a specific column in a table.
• Record – applies the function to a specific row in a table.
• Table – applies the function to the entire table.

Alternate Method for Help on a Specific Function

If you have a casual curiosity about how a specific function operates, you can type the name of the
function into the Formula Bar, press ENTER, and view the help for that function.

TABLES, LISTS, & RECORDS – HOW TO REFERENCE THEM IN A TABLE
Lists
Think of a list as a table with only one column. What makes the list different is that you can have a
mixture of data types within the same single column.
When working with lists, Power Query provides a new set of tools in a second Transform ribbon under
List Tools. This replaces the functionality of the traditional Transform ribbon. (Note the different icon in
the queries list.)
Creating a List from a Table

Creating a list from a table can be accomplished by selecting the desired column and either:
• selecting Transform (tab) → Any Column (group) → Convert to List, or

• right-clicking the column heading and select Drill Down.
Referencing Column Headers in Lists

It is a tradition in the world of databases to enclose field names/column headers in square brackets.
This informs the database of the entire field name even if there are embedded spaces.
[Employee Name]

Power Query uses the same nomenclature.
The above list was generated from a column named “Employee Name” from the previous step named
“Changed Type”.
Records
Think of a record as a table with only one row. When you click on a row number in a table, the row is
displayed in the lower panel as a record.
Each value on that row is associated with a field/column in the table.
Referencing Tables, Lists, Records, and Cells

Suppose we have a query named “TEmployees” and appears as follows.
From another query (or step), type the following for the various references:
Reference the Entire Table
= TEmployee

Reference a Specific Field (column)
= TEmployee[Employee Name]
Column names are always enclosed in square brackets.
Reference a Specific Record (row)
= TEmployee{0}
Row numbers are always enclosed in curly braces.
Remember; Power Query begins counting from 0 (zero).

This also provides access to the Row Tools → Convert ribbon by which you can convert a record into a
table.

Reference a Specific Value (cell)
= TEmployee[Employee Name]{0}
This also provides access to the Text Tools → Transform.
NOTE: When referencing values, the order of Field/Row can also be made as Row/Field. Most users
prefer the Field/Row version.
= TEmployee[Employee Name]{0}
or
= TEmployee{0}[Employee Name]
Performing a Search on a Table

If you want the record retrieval to be more dynamic, you can use comparative operations within the
reference.
Instead of hard-coding the record position,
=TEmployee{5}
…we could search for a record that meets defined criteria.
=TEmployee{ [Employee Name = “West Kim”] }

NOTE: This only works if there are no duplicates in the requested field. If there are duplicates, an error
will occur.
Returning a Value from a Searched for Record

Suppose we wanted to return the salary for Kim West. The syntax would be as follows:
=TEmployee{ [Employee Name =”West Kim”] } [Salary]
Returning a Table from a Table

If you want to return multiple columns from a table, enclose the column names (separated by commas)
within an extra set of encompassing square brackets.
=TEmployee[ [Department], [Salary] ]
This will produce a reduced version of the original table.

BRACKETS & LOOKUP OPERATORS IN M CODE

Defining the Character
• Brackets ( ) ...................... also known as parentheses

• Square Brackets [ ]
• Curly Brackets { } ............ also known as braces
Functions use Brackets
The brackets enclose all the arguments for the function.
Text.Contains( “Finance Manager”, “Manager” )
Fields use Square Brackets

Fields (also known as column headings) use square brackets to allow for spaces in the field names.
Text.Contains( [Stock Symbol], “MSFT” )
The above example would return all rows that have “MSFT” as the stock symbol.
Square brackets can also be used during field selection to return a single record.
Text.Contains( [Employee Name = “West Kim” ] )
When defining several fields, each field is enclosed in square brackets while the entire set of fields is
enclosed in a set of square brackets.
[ [Employee Name], [Position] ]
Positional Operators use Curly Brackets

When referencing a position (row) in a list or table, the position is enclosed in curly brackets.
The following example would return the 3rd row from the results of the “Changed Type” step.
#”Changed Type” {2}
Curly brackets can also hold a list of values. These values can be of differing data types.
{ 1..100 }
{ “Department”, 500 }

3 Important Power Query Rules

1. Power Query is Case Sensitive
2. Power Query is Type Sensitive
3. Power Query is zero-based
Query’s Result:
A Query’s result is what you see in the last step of the query. This is what the query returns.
Step Identifiers:
If step names have spaces in the name you get the hash sign and quotation marks around the name. For
example: #”Changed Type”.
If step names don’t have a space in the name, you can directly reference the name: ChangedType.

CREATING LISTS, TABLES, & RECORDS –

TO USE AS PARAMETERS OR TESTING OF FUNCTIONS
One way to test column manipulations without risking corrupting your existing query is to start a new
blank query and extract a column from the other query as a list.
= #”Changed Type” [Department]
You can now practice the manipulations without fear of damaging your existing work.
Creating a List from Scratch

There are several types of lists you can create. A list of items can be created using the following syntax
rules:
• The entire list is enclosed in curly brackets

• Text items must be surrounded by double-quotes
• Items are separated by commas
={ “Australia”, “Europe”, “United States”, “Asia” }

Also, a list of numbers can be created but with shortcut syntax.
={ 1 .. 10 }
This shortcut syntax extends to the alphabet as well.
={ “a”.. “z” } & { “A”.. “Z” }

Lists can contain a mixture of data types.
={ “Africa”, 100, “South America”, 200 }
List of Lists
It sounds strange, but a list can contain other lists.
={ { “a”.. “z” }, { “A”.. “Z” } }
This is nothing new; you’ve been using “lists of lists” since day one of using Power Query.
Examine the following #”Changed Type” step:
It’s a list of lists. In this case, column names and data types.

Creating a Record from Scratch

Records are created using square brackets.
=[Company ID = 123]
The name of the column “Company ID” is not enclosed in square brackets because the entire statement
is within square brackets.
To create a record with multiple columns, the syntax is as follows:
=[Company ID = 123, Company Name = “Bold”]
Creating a Table from Scratch

Creating a table from scratch requires the use of a table function.
= #table()
The above table function requires a list for the column headers and a list of lists for the data.
= #table( { “Company ID”, “Company Name” }, { { 123, ”Bold” }, { 234, “Lucas” } } )

UNDERSTANDING THE EACH KEYWORD –

AND THE PURPOSE OF THE UNDERSCORE
At its core, the Each keyword tells Power Query to loop through many values, usually rows.
This would be analogous to writing a single formula in Excel and using Fill Series to repeat the formula
down a column for a series of rows.
Consider the following instruction:
=Table.SelectRows(#"Changed Type", each ( [Department] = "Sales" ) )
We’re using the Table.SelectRows function to extract records where the [Department] is equal to
“Sales”.
The function will process each row using the same criteria.
Shorthand Notation
The Each keyword is shorthand notation for a function and has a single argument. This allows us to
write a custom function within a step without confusing syntax.
Examine the following code:
= Table.SelectRows(#”Changed Type”, (myRule) => (myRule[Department] = “Sales”) )
We have an embedded function within the step.
We can simplify the code by replacing this blue part…
… with the word each.
= Table.SelectRows(#”Changed Type”, each ([Department] = “Sales”) )
Underscore Notation/Reference
There will be steps that utilize an underscore character in their code.
= Table.TransformColumns(#"Filtered Rows",
{ {"Employee Name", each Text.BeforeDelimiter( _, " " ), type text } } )

The underscore character is M Code for “element in current column”. Since we have defined the
“Employee Name” column as an argument for the Table.TransformColumns function, we don’t need to
declare it a second time for the Text.BeforeDelimiter function. The underscore denotes elements in the
already defined column.
Here is another example of the use of an underscore character when working with lists:
= List.Select(#"Employee Name", each _ > "M")
The underscore relieves us from writing a complicated reference to elements in an already declared
source.

USING POWER QUERY PARAMETERS

Parameters are variables that are defined at query execution time to influence or alter the steps of a
query.
Some examples of parameters include:
• Determine the location of the source file

• Prompt for an argument from the user, like a discount rate
• Prompt for arguments from the user to apply as filters (ex: country and sales rep)
• Prompt for a column to sort and by what method (ascending or descending)
If we were to create a query that performs a series of transformations, the processing rules are the same
each time the query is refreshed.
In this example, the #”Filtered Rows” step is hard-coded to “America”.
Each time we want to run a query for a different Region, we must either modify the existing query or
(worse yet) write an entirely new query that is mostly the same.
Suppose we could write the step in a way that prompts the user to pick a region each time the query is
refreshed.

In the above case, the reference to SelectRegion is directed to another query.
The code behind the SelectRegion parameter is as follows:
Creating Managed Parameters

To create a Managed Parameter, select Home (tab) → Parameters (group) → Manage Parameters →
New Parameter.
The Managed Parameters dialog box allows us to create, modify, and delete parameters.

The fields are as follows:
• Name – define a name for the parameter

• Description – provide a brief overview as to the purpose of the parameter
• Required – define whether the parameter is required or optional
• Type – the data type stored by the parameter
• Suggested Values – define whether the parameter will accept any value, a value from a pre-
defined list, or another query
• Default Value – the default selection if not chosen by the user
• Current Value – the currently selected value
Update the Query to Use the Parameter

Once the parameter has been created, modify the query step that is hard-coded by replacing the static
value with a reference to the parameter.
becomes
Using the Parameter

Once the parameter is created, you can set the value of the parameter by selecting the parameter query
and entering the needed value.
The moment you switch back to the query that calls the parameter; the results have changed.

If you have several parameters that need to be set, you can select Home (tab) → Parameters (group) →
Manage Parameters → Edit Parameters to launch a dialog box where all parameters are displayed in a
convenient interface.

SPEED UP QUERIES – TABLE.BUFFER & HOW TO TEST IMPACT

When you reference another query from the existing query, Power Query will have to refer to it and
load what it needs. This is fine if the reference to the other query is once or just a few times. It can
however get quite slow if you are referencing another table in the “Add Custom Column” tab because
that table will be referenced for EVERY SINGLE row.
One trick is to force Power Query to read the entire table into memory before step execution. This way
the other table will be read only once instead of for each single row.
This is where the Table.Buffer function comes into play.
You can add the Table.Buffer code to the table you want loaded once as shown below.
This concept doesn’t just apply to “other” queries, but also when you reference a previous step of the
current query and you’d like that step (the table) to be loaded once in memory.
Before:
After:

This can be especially useful when passing a table to a function as the table will be read only once.
Why Is This Behavior Not the Default?

The Table.Buffer function will break query folding (see next topic) as the full table resides in local
memory. If you are reading against a database (i.e. SQL), you can no longer pass responsibility for
following steps to the server via query folding.
Testing Performance On Large Data Sets

It is not a safe assumption to think that a query with a few steps will run faster than a query with many
steps. The determining factors in terms of performance are the features being used during the steps.
It is advised to use the interface features and create many steps as opposed to writing a custom step
that reduces many steps into a single step.
It is almost impossible to test performance on a small data set. The real test for a query’s performance
is when you get into the hundreds of thousands or millions of records.
To keep from creating a large dataset, we can replicate our existing data as many times as we like using
a Table function.
The function is named Table.Repeat and its only arguments are:
• table to be repeated
• number of times to repeat
An example of its use would be the following:
= Table.Repeat( Source, 800 )
We can now run any number of transformations against the large data set to test performance.
Once testing is complete, we can delete this instruction and the remaining transformations will now
execute against the actual data.
Counting the Rows in a Data Set

If you select a query step and activate the profiling feature (View (tab) → Data Preview (group) →
Column Profiling), you can see from the Count statistic the number of rows in the data.

After we have used the Table.Repeat function, we see a noticeable increase in table rows.
NOTE: If you are only seeing 1000 as the Count result, you likely need to change the option to view the
entire data set (bottom-left corner).

QUERY FOLDING – IMPROVE PERFORMANCE FOR RELATIONAL DATABASES
What is Query Folding?

Whenever possible, Power Query translates transformation steps to the native language of the data
source system.
This shifts the responsibility of data manipulation to the source system (i.e. an SQL server, Exchange
server, or OData server). The more work the source machine can do as opposed to your local machine,
the faster the queries process. Plus, potentially less data needs to be sent from the source to the
workstation.
Imagine a scenario where you have a 10 million record data set of sales by month for one year. You
want to show the total sales by month. You could download all 10 million records, group by year, and
aggregate the sales.
Query folding would shift the grouping and aggregating of data to the server, thereby only delivering 12
records to the user.
NOTE: Query Folding does not apply to flat files like Text files or Excel files.
Viewing Query Folding Instructions

To view the Query Folding instructions that are sent to the server, right-click on a query step in the
Applied Steps list and select View Native Query.
This will display the code necessary to perform the selected step in the source language.

As you progress down the list of query steps, you can view each new transformation’s instructions.
Some steps may contain very little code while other steps can get quite lengthy.
NOTE: Displaying the native query is not always possible for all sources, even when query folding is
occurring. Connecting to an OData feed will capitalize on query folding, but you will not be able to
display it using the View Native Query feature. To see the query folding operations, you would need to
consult the OData server log files.
Breaking Query Folding

Once you perform a manipulation on the data that cannot be offloaded to the source server (ex:
replacing values in a column or adding an index column), all steps from that point forward MUST be
performed locally even if a later manipulation is one that is normally accepted as a server-side feature.
You will want to develop your queries where you perform as many server-side manipulations to the data
before performing any client-side manipulations.

IF THEN STATEMENT –
LOOKUP OPERATORS TO GET VALUE FROM PREVIOUS ROW
Creating Row Against Row Comparisons

In a previous lecture (Section 15 – Lesson 4) we learned how to calculate the difference in price from the
current row against the price in the previous row.
This was performed by merging the table with itself using two index columns.
Now we will see how to perform the five steps needed previously with a single IF Then statement.
1. Delete all query steps AFTER the creation of the first index column.
2. Insert a Custom Column.
3. Name the new column “Difference”.
4. The custom formula is as follows:
= if [Index] = 0 then null else [Gold Price Close] - #"Added Index"[Gold Price Close]{[Index] - 1 }
The logic breaks down as follows:

If the [Index] on the current row is a 0
then place a null in the result, otherwise
take the current row’s [Gold Price Close] and
subtract the previous row’s [Gold Price Close]
The tricky part is in the last step.
[Gold Price Close] - #"Added Index"[Gold Price Close]{[Index] - 1 }
When we reference [Gold Price Close], we are looking at a single value for the current row. This changes
as we move from row to row during table processing.
We need to point to the entire column of gold prices, so we reference the previous query step’s results
as a reference table.
#"Added Index"[Gold Price Close]
We need to locate a specific gold price from this column, so we use the value stored in [Index].
#"Added Index"[Gold Price Close]{[Index]}
Because Power Query starts counting rows at 0 (zero), we need to make a small adjustment to the value
by deducting 1 from the [Index].
#"Added Index"[Gold Price Close]{[Index] - 1 }
This returned value is subtracted from the [Gold Price Close] of the currently examined row.
[Gold Price Close] - #"Added Index"[Gold Price Close]{[Index] - 1 }
The result is as follows:

ERROR HANDLING IN POWER QUERY –

BULK REPLACE LOOKUP WITH TRY OTHERWISE
Testing the Waters

In a previous lecture (Section 15 – Lesson 3) we learned how to perform a bulk replacement of App
names by merging two tables.
This was performed by merging the two tables and creating a Conditional Column to extract the
appropriate App name.
Now we will see how to perform this same operation with a single Try Otherwise statement.
The Purpose of Try Otherwise

The purpose of Try Otherwise is to perform a manipulation and see if it succeeds or fails. If it succeeds
then we are delivered our result. If it fails, we can implement a separate action like display a message or
perform a completely different manipulation.
If you have ever used Excel’s IFERROR (or the older IF(ISERROR) combination), the principle is the same.
Creating the Custom Column to generate New App Names
1. Create a reference to the Data query.
2. Insert a Custom Column.
3. Name the new column “New App”.
4. The custom formula is as follows:


Go to the query named TransformTable and locate the current row’s [App] in the [Before]
column.
Once located, return the corresponding app name from the [After] column.
The results are as follows:
= TransformTable{[Before = [App]]}[After]
Although we are retrieving the new app names for listed apps, we are receiving errors for apps that do
not require new names.
Open the Custom Column and modify the formula as follows:
= try TransformTable{[Before = [App]]}[After] otherwise [App]


Try to go to the query named TransformTable and locate the current row’s [App] in the [Before]
column.
If the app is located, return the corresponding app name from the [After] column.
Otherwise, return the app name from the [App] column of the current table.
Bonus Step
The next logical step would be to change the data type of the New App column to Text.
We can integrate this data type change into the Custom Column’s logic.
1. Select the Added Custom step in the Applied Steps list.
2. Add the following instruction to the end of the M Code in the Formula Bar:
, type text
… becomes …
= Table.AddColumn(Source, "New App", each try TransformTable{[Before = [App]]}[After]

otherwise [App], type text)

Part 12
Working with
Lists & Table
Function
POWER QUERY TEXT FUNCTIONS
Text.Contains, Text.Replace, etc.

We have constantly promoted the use of the button interface tools to perform data manipulation.
Unfortunately, many manipulations don’t have representation on the Ribbon. This is where you need to
write custom formulas using a function from the Power Query M function library
https://docs.microsoft.com/en-us/powerquery-m/power-query-m-function-reference
No one can become an expert with every function in the library, but if you take some time to browse
certain categories of interest, like Text functions, you may discover some hidden jewels that are worth
investigating.
Many of these functions are like Excel functions. You may only need to learn the new name for the
function and you’re ready to use it in a formula.
Excel Power Query

LEFT Text.Start
RIGHT Text.End
LEN Text.Length
PROPER Text.Proper
REPLACE Text.Replace
FIND Text.PositionOf
Power Query includes many helpful modification functions, such as:
• Text.Remove
• Text.Insert
• Text.Replace
Text.PositionOf
The syntax for Text.PositionOf is
Text.PositionOf( text as text,

substring as text, optional
occurrence as nullable number, optional
comparer as nullable function)
as any

Power Query
Example of Text.PositionOf
Text.PositionOf(“May the Force be with you”, “Force”)
The result would be 8 because the word “Force” starts in the 8th character position of the text.
Remember, Power Query starts counting at 0 (zero).
Practical Example
We have the following data:
We need to perform the following manipulations:
• Determine which Article Descriptions contain the phrase “simple”

• Remove the colors from the Article Descriptions
Locating the Phrase “Simple”
This example utilizes the Text.Contains and Text.Proper functions.
Create a Custom Column using the following formula:
= Text.Contains(Text.Proper([Article Description]), "Simple")

The formula takes the [Article Description] and capitalizes the first letter of each word using the
Text.Proper function.
It then locates the phrase “Simple” in the [Article Description] using the Text.Contains function.
We will data type the output as “True/False” responses by adding the proper declaration to the end of
the M Code in the Formula Bar.
type logical
Removing the Color from the Article Description

This example utilizes the Text.Replace and Text.Proper functions.
Create a Custom Column using the following formula:
= Text.Replace(Text.Proper([Article Description]), " (" & Text.Proper([Color] & ")"), "")
The formula takes the [Article Description] and capitalizes the first letter of each word using the
Text.Proper function. We use the same function to capitalize the first letter in the [Color] field.
Using those results, we concatenate a “space-open parentheses” to the [Color] to a “close parentheses”.
“ (“ & Text.Proper[Color] & “)”

Power Query
The Text.Replace function locates the concatenated result in the [Article Description] and replaces it
with nothing (two double quotes).
We will data type the output as Text responses by adding the proper declaration to the end of the M
Code in the Formula Bar.
type text

Learn More
More Resources

Power Query
MORE RESOURCES
In addition to this Reference book, we’ve also put together a Power Query Solution book for you to
handle different scenarios you might come across. This book is also available as a resource inside your
online Power Query course on XelPlus.com.
Let's talk about more learning.
On my site, you’ll find weekly Excel articles on different aspects of Excel: XelPlus.com/tutorials/
There are a lot of great Excel websites out there. For more Excel information, check out my list of
resources here: XelPlus.com/resources
The more you know of Excel the easier it gets for you to implement complex projects.
If you're interested to become more advanced and create great visualizations or effective dashboards,
check out my other courses: https://www.xelplus.com/courses.
I also share weekly tutorial videos on my YouTube channel so make sure you check it out if you're
interested to learn more. Just type in Leila Gharani on YouTube.
I wish you the best of luck on your journey and hope to see you in my other courses.

Power Query - Reference Book PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Power Query - Reference Book PDF

Uploaded by

Copyright:

Available Formats

Master Power Query Power Query – Reference Guide

Copyright © 2020 Leila Gharani

Refresh Data and Refresh Options................................................................................................. 21

Delete a Query ..................................................................................................................................... 45

Working with Filter – AND & OR Conditions .................................................................................. 75

Examining the Unpivot Query ........................................................................................................... 103

Copyright © 2020 Leila Gharani

Other Intrinsic Functions ................................................................................................................... 135

Copyright © 2020 Leila Gharani

Creating a Compound Key ................................................................................................................. 158

Copyright © 2020 Leila Gharani

Using a Function in a Query ............................................................................................................... 180

Query Folding – Improve Performance for Relational Databases ................................................. 209

Copyright © 2020 Leila Gharani