TODAY's MENU

HERE WE SERVE POWER QUERY M.

OPTIMIZE TYPE SETTING, PART 2

TYPE CONVERSION

DID YOU
KNOW?
Introduction
In the first part, we talked about ascribing data types - defining a
column's data type without conversion or type check (until runtime).
Now, we'll explore retaining data types and type conversion in M.

Here's a pro tip: if you make "Change Type" the final step in your query,
you can limit the number of steps required. Most users don't do this
because Table.TransformColumnTypes doesn't have a MissingField.Type
parameter causing the query to break when a column doesn't exist.

I recommend to disable the setting: Automatically detect column types and

headers for unstructured sources and use Transform/ Detect Data Type with
care. These only scan the top 200 rows to determine type, therefore
always need to be validated.

Let's find out how and where we can retain type to reduce steps and
explore different methods to convert type.
Losing type after invoking Table.ReplaceValue
When using Transform/ Replace Values you can match entire cell content
using Replacer.ReplaceValue or look for each occurrence of the oldValue
in a cell with Replacer.ReplaceText. The former will return type any,
the latter can retain a declared type text, as shown here.

Above type text reverts to type any. While below, type text is retained.

However, keep in mind that Replacer.ReplaceText will only retain type

text if the column values were already declared type text; otherwise,
it will return type any.
Losing type after expanding a nested table
Expanding a column with a “type table” does not garantee retaining
nested table column-types.
Types are retained when specified in the outer table type, shown here:
let
Source = Table.FromColumns(
{
{"A","B"},
List.Repeat( { #table( {"Col1", "Col2"}, {{"1",1}, {"2", 2}}) }, 2)
}, type table [ColA=text, ColB= table[Col1=text, Col2=number]]
),
ExpandCol2 = Table.ExpandTableColumn(Source, "ColB", {"Col1", "Col2"})
in
ExpandCol2
Quickly restore types for an entire table
To restore data types to a previous state in your query, you can use this
simple piece of M code.
Value.ReplaceType( PrevStepName, Value.Type( #"Changed Type" ))

The table shape, provided to these functions, needs to match or an error

is returned. This will revert the type back to the #"Changed Type" step
for all columns in the PrevStepName table.
Exploring changing type through the UI
When changing a column's type, Table.TransformColumnTypes is called,
which performs a type conversion to the primitive nullable type.
An error is raised when a value can't conform to the selected primitive
type, or when the column doesn't exist.

Using the user interface for this transformation generates a list with
typeTransformation lists in the following format:
{column name, type name}

However, the hard-coded column names in combination with the

absence of a MissingField.Type parameter make this an error-prone
transformation.
 Type conversion matrix

https://learn.microsoft.com/en-us/power-query/data-types#data-type-conversion-matrix
Leverage a naming convention to set type.
To make the "Change Type" step less error-prone, avoid hard-coding
column names. Instead, leverage your naming convention for more
precise results.
Table.TransformColumnTypes( PrevStepName,
List.Transform(
List.Select( Table.ColumnNames(PrevStepName),
each Text.EndsWith( _, " PCT", Comparer.OrdinalIgnoreCase)
), (x)=> {x, Percentage.Type}
)
)

The code above will convert all columns in the table where the column
name ends with "PCT" (ignoring case) to a Percentage.Type.
You can use the combination operator (&) to add more sets.

Note that Table.TransformColumnTypes can't assign types to more

complex values like: lists, records, tables, or functions. In such cases,
you'll need to use Table.TransformColumns.
The amazing Table.TransformColumns
Marcel Beugelsdijk developed the core part of this solution. The following
code examines the first non-null value in each column, determines its
type, and sets it as the column type, whether primitive or structured:

Above we're starting with a list of column names and transform each
name into a transformOperations list with this format:
{ column name, transformation, new column type }

The _ (underscore) represents the column name. each _ won't cut it as

transformation, Marcel's work around is to place the current value in a list
and use item access to retrieve it again {_}{0} Finally, return the type of
the first no-null value from the column with the help of Value.Type.

If your columns don't contain mixed values, this is a dynamic

method to assign column types without the default 200 row limit.
Leverage colNames to take it one step further
Naming conventions help to create dynamic logic when dealing with data
types. Cameron Wallace's defaultTransformation trick is a great example.
Table.TransformColumns( PrevStepName,
let colNames = Table.ColumnNames( PrevStepName ) in
List.Transform( List.Select( colNames, each
Text.EndsWith( _, "Price", Comparer.OrdinalIgnoreCase)),
(x)=> { x, each {_}{0}, Currency.Type} ) &
List.Transform( List.Select( colNames, each
Text.Contains( _, "Name", Comparer.OrdinalIgnoreCase)),
(x)=> { x, each {_}{0}, type text} ),
Number.From
)

The _ (underscore) represents the column name, as transformation place

the current value in a list and use item access to retrieve it again {_}{0}
Finally, provide a type for that set like Currency.Type or type text.
Use the ampersand (&) to combine more sets and use the optional
defaultTransformation to change type for all remaining columns, shown
here, into a nullable number, the return type of Number.From
Note that the most common and safe defaultTransformation
would be Text.From to convert to a nullable text.
Conclusion
I trust these examples will make working with data types a breeze.
In Part 1, we learned about ascribing data types, where we define a type
without conversion or type checking until runtime. Here, in Part 2, we've
dived into retaining data types and various ways to convert type.

It's important to understand that there are always two types at play:
value-type and column-type. Until column-type is set, you are dealing with
the value-type. A column with numeric values may be of a text type,
leading to errors in calculations that require a number type. To avoid
these errors, use one of the conversion functions, listed here:

https://learn.microsoft.com/en-us/powerquery-m/type-conversion

Keep in mind that dealing with data types in Power Query requires
careful attention and knowledge. By following these tips and tricks,
you can work with data types like a pro.

Like my content?
Consider a connection,
following me or #pqtodaysmenu
 I LOVE SOLVING
PRACTICAL PROBLEMS
WITH POWER QUERY/ M.

Melissa de Korte pqtodaysmenu