Post on Tony McGovern. Hacker. Artist. Storyteller. Data lover.

Handling Unsuccessful HTTP Requests in Power Query M

Wed, 16 Jan 2019 00:00:00 +0000

Many web APIs implement a standard set of HTTP response codes that indicate whether a request was successful. A request that returns expected data often comes with a successful 200 response code. Perhaps the most well-known response code is 404, which tells us a web page doesn’t exist. When I request data using the Web.Contents Power Query M function in Excel or Power BI and that request is unsuccessful, I can often make sense of the response code that comes with it to quickly fix my query. Most other times, I encounter unfamiliar response codes that take me far too long to resolve, if I resolve them at all.

Inspired by a thread on a Power BI User Group forum, I’d like to share an error handling pattern I’ve recently adopted that helps me quickly resolve unsuccessful HTTP requests using Web.Contents. This pattern consists of two parts: a) using the ManualStatusHandling option in the Web.Contents function to catch likely HTTP response codes, and b) creating error tables that explain to the user the likely reasons an HTTP request fails. With this error handling pattern in place, my queries are now more resilient and provide a better user experience.

So how does it work? Suppose you’re interested in the latest population estimates for every state in the United States from the U.S. Census Bureau’s API. A successful request returns the expected data along with a 200 response code:

let
    response = Web.Contents(
        "https://api.census.gov/data/2018/pep/population?get=POP&for=state"
    ),
    json = Json.Document(response)
in
    json

If I misspell the URL and instead send this request:

let
    response = Web.Contents(
        "https://api.census.gov/data/2018/BADSPEL/pupuracion?get=POP&for=state"
    ),
    json = Json.Document(response)
in
    json

I get a 404 response code, which tells me the page does not exist.

But what happens when I misspell another part of the URL instead, say one of the parameters?

let
    response = Web.Contents(
        "https://api.census.gov/data/2018/pep/population?get=POP&for=states"
    ),
    json = Json.Document(response)
in
    json

I now get a 400 response code:

Why is this? Well, the first unsuccessful query told me I misspelled the location of the resource (the portion of the URL before the ‘?’) whereas the second unsuccessful query told me I misspelled one of the paramters in the URL. While this is useful information for me, the Power Query M developer, these sorts of subtleties are lost on an end-user. It’s likely your user won’t care to know the difference between a 400 and 404 response code; they just want to know why their request failed. (There’s an extra ’s’ in the word ‘state’, in case you were wondering why the second request failed.)

So how does this relate to error tables? Like most well-documented APIs, the U.S. Census Bureau API has a page devoted to listing and describing all the possible response codes that can be returned by their service. I take this information and build an internal table within the query that defines and describes these response codes in my own words. I’m now able to throw custom messages that make the difference between a 400 response code and a 404 response code more obvious.

For example, in the code below, I use the Error.Record function to create individual records that allow me to catch these unsuccessful requests and throw my own custom error messages to the user. I then create an extra field in each record called ‘Status’, which maps each HTTP response code returned by the API to a corresponding error message of my choosing:

let
    // define possible errors
    errors.badRequest = Record.AddField(Error.Record("Bad Request", "this dataset does not exist. It's most likely a misspelled parameter. Double check the spelling of the geography and variables you're passing as parameters."), "Status", 400),
    errors.invalidAccessKey = Record.AddField(Error.Record("Invalid Access Key", "Make sure the provided access exists and is entered correctly."), "Status", 401),
    errors.insufficientPermissions = Record.AddField(Error.Record("Insufficient Permissions", "you don't have the right permissions to access this dataset."), "Status", 403),
    errors.resourceNotFound = Record.AddField(Error.Record("Page Does Not Exist", "this dataset does not exist. It's most likely a misspelling somewhere in the URL. Double check the URL and make sure it's spelled correctly."), "Status", 404),
    errors.tooManyRequests = Record.AddField(Error.Record("Too Many Requests", "you've reached your daily limit. Come back in 24 hours and try again."), "Status", 429),
    errors.backendError = Record.AddField(Error.Record("Server Error", "there is an unknown server error preventing this request from being executed. You've done nothing wrong. Just try again in a bit."), "Status", 500),
    errors.serviceUnavailable = Record.AddField(Error.Record("Service Unavailable", "the server is too busy right now and needs a short break. Don't worry, you can try again in a bit."), "Status", 503),
    // build an error table
    errors.table = Table.FromRecords({errors.badRequest, errors.invalidAccessKey, errors.insufficientPermissions, errors.resourceNotFound, errors.tooManyRequests, errors.backendError, errors.serviceUnavailable})
in
    errors.table

I then combine this with the ManualStatusHandling option in Web.Contents, where I list all the HTTP response codes provided by the API. So, if the API returns one of those response codes, ManualStatusHandling tells my request to fail silently, thus allowing me to trap the unsuccessful request in a subsequent step. Chris Webb has a great explainer on what the ManualStatusHandling option does and provides examples on how to use it.

With both an error table and the ManualStatusHandling option implemented, my query now handles unsuccessful API requests. Here’s an example where I throw a custom error message to handle a 404 response code (the ‘state’ parameter still has that extra ’s’):

let
    // define possible errors
    errors.badRequest = Record.AddField(Error.Record("Bad Request", "this dataset does not exist. It's most likely a misspelled parameter. Double check the spelling of the geography and variables you're passing as parameters."), "Status", 400),
    errors.invalidAccessKey = Record.AddField(Error.Record("Invalid Access Key", "Make sure the provided access exists and is entered correctly."), "Status", 401),
    errors.insufficientPermissions = Record.AddField(Error.Record("Insufficient Permissions", "you don't have the right permissions to access this dataset."), "Status", 403),
    errors.resourceNotFound = Record.AddField(Error.Record("Page Does Not Exist", "this dataset does not exist. It's most likely a misspelling somewhere in the URL. Double check the URL and make sure it's spelled correctly."), "Status", 404),
    errors.tooManyRequests = Record.AddField(Error.Record("Too Many Requests", "you've reached your daily limit. Come back in 24 hours and try again."), "Status", 429),
    errors.backendError = Record.AddField(Error.Record("Server Error", "there is an unknown server error preventing this request from being executed. You've done nothing wrong. Just try again in a bit."), "Status", 500),
    errors.serviceUnavailable = Record.AddField(Error.Record("Service Unavailable", "the server is too busy right now and needs a short break. Don't worry, you can try again in a bit."), "Status", 503),
    // build an error table
    errors.table = Table.FromRecords({errors.badRequest, errors.invalidAccessKey, errors.insufficientPermissions, errors.resourceNotFound, errors.tooManyRequests, errors.backendError, errors.serviceUnavailable}),
    // get response
    response = Web.Contents(
        "https://api.census.gov/data/2018/pep/population?get=POP&for=states", 
        [
            ManualStatusHandling = {400,401,403,404,429,500,503}
        ]
    ),
    responseMetadata = Value.Metadata(response),
    responseCode = responseMetadata[Response.Status],
    responseHeaders = responseMetadata[Headers],
    json = if responseCode <> 200 then error errors.table{List.PositionOf(errors.table[Status],responseCode)} else Json.Document(response)
in
    json

This error handling pattern provides a more resilient query that catches most HTTP response codes I’ll likely encounter. Moreover, I get to describe these responses in my own words, allowing me to better communicate to my users unsuccessful Web.Contents requests.

What error handling techniques do you use in Power Query M? Comment below – I’m interested to hear about your patterns.

Map of Wisconsin midterm election results for 2018

Wed, 12 Dec 2018 00:00:00 +0000

Wisconsin Midterm Election Results for 2018

When voters in the state of Wisconsin cast ballots for national and state elections, they do so at the “ward”-level, an administrative boundary created to conduct elections. Yet when it comes to tabulating and reporting election results, Wisconsin officials combine wards into “reporting units”, larger administrative boundaries comprised of one or more wards. But there is no authoritative body that produces shapefiles for these reporting unit boundaries, leaving analysts, journalists, and the public little opportunity to visualize Wisconsin election results in any meaningful way.

For the 2018 midterm elections, John Johnson, Research Fellow at the Marquette Law School, filled in that gap and created a shapefile that provides high-quality, consistent boundaries and election results for Wisconsin. Expanding on his work, I used his shapefile – which also contains election data – and visualized the results.

I created the map in Power BI using the Mapbox Visual. I converted the shapefile to geojson with mapshaper and simplified its features using mapshaper’s command line tools. I applied the Visvalingam simplification method through the following statement:

-simplify 10%

I converted the geojson into a tileset using Mapbox Studio.

Power Query M Version Control using GitHub

Mon, 12 Nov 2018 00:00:00 +0000

Power Query M Version Control using GitHub

There is currently no obvious version control process to see a history of changes made to Power Query M functions within Excel or Power BI. The Advanced Editor in the Power Query Editor has no built-in function to capture changes made to the query. In other words, it’s very difficult to roll back queries to earlier versions once they’re executed.

I tend to have a fairly involved data preparation process that I build in to my Power Query M functions. It’s not uncommon for my queries to contain hundreds of lines. Once they become this large, managing changes to my queries starts to become onerous. It’s at this point I often switch from writing Power Query M functions in the Advanced Editor to writing them in a text editor like Notepad++. One of the benefits of separating the functions from the Power BI or Excel file is that I can store the functions in a .pq file on GitHub and see the entire timeline of changes made to the file. Storing .pq files on GitHub becomes especially useful in collaborative envrionments where team members work on the same set of Power Query M functions. And even if you’re working by yourself, GitHub is still useful for those projects that contain a number of queries, like this library of M functions that prepare data from the Socrata Open Data API, for example.

So if I’m not writing these functions in the Advanced Editor, how does the Power Query Editor know they exist? How do we execute Power Query M functions that reside somewhere else?

Expression.Evaluate() and the `#shared` Environment

Consider the Power Query M code below:

let
  Source = 
    Expression.Evaluate(
      Text.FromBinary(
        Web.Contents(
          "https://raw.githubusercontent.com/tonmcg/powersocrata/master/M/Socrata.ReadData.pq"
        )
      ),
      #shared
    )
in
  Source

If we copy the code into the Advanced Editor, we should see the following function appear:

The screenshot shows the Power Query Editor rendering a function with multiple parameters. Not anywhere in the code above have we defined parameters. So where did they come from? The answer is I created a custom Power Query M function called Socrata.ReadData() and stored it in a repository on GitHub. I’m using the Web.Contents() function to return the contents of the file and then the Text.FromBinary() function to render the text contained within. The parameters are defined within the function itself, which again, is stored on GitHub.

The most interesting part of the function is the use of Expression.Evaluate(). At its simplest, it evaluates text and returns an evaluated value. In our case, the evaluated text is the text within the Socrata.ReadData() function. The evaluated value is the actual function itself. Put another way, Expression.Evaluate() allows us to make a call to GitHub and return the function’s contents, render the text of the function, and tell the Power Query engine to treat the rendered text as an M function.

But that’s only half of the answer. While the first parameter of the Expression.Evaluate() function requires an expression as text, the second parameter asks for the environment in which the text resides. In our example, I’ve defined the envrionment as the #shared variable. This pattern is what allows a Power Query M function that resides on GitHub to be executed in our Power Query Editor.

I’ll not go into it here, but if you’re interested in understanding the environment parameter and the #shared variable, here is a list of resources that should get you started:

Lars Schreiber and Imke Feldmann have done incredible work simplifying these concepts
Chris Webb provides simple examples that help motivate understanding in this area
Finally, Imke Feldmann uses this pattern often and provides numerous examples on her blog. Here’s one example.

Making it Work in Power BI Dataflows

The most exciting part, however, is that this pattern works not only in Excel and PowerBI^[1], but through the browser as well with Power BI Dataflows. We can get the Socrata.ReadData() function to work in dataflows by using the Power Query M code above in the Power BI dataflows Query Editor:

If you want to see what this looks like in action, I tweeted a GIF of this working in Power BI dataflows.

There are boundless opportunities here to use GitHub’s version control system to manage the entire timeline of changes made to projects that use Power Query M functions. Distributed teams around the world can simultaneously work on the code, understand what changes have been made, and collaborate at any time while maintaining source code integrity.

Do you use this pattern as well? What else have you found? Let me know in the comments section below.

Update 13 November 2018

Matt Masson offers a comment in response to this blog post:

Well the gateway infrastructure doesn't support dynamic data sources, and I'm not sure if / when that will happen. I'm also not sure it is wise to ever pull dynamic code from web locations.
— Matt Masson (@mattmasson) November 12, 2018

I think Matt is suggesting this pattern of storing M code on a separate server could open up your application or dataflow to an exploit. If so, he’s absolutely correct. The code above is stored on a public GitHub repository, which means anyone can submit a pull request and sereptitiously inject code that can cause a security vulnerability.

This solution is far from ideal. But the question remains: is there a secure version control system that allows me to see changes to my M code over time? If you have ideas, feel free to leave them in the comments below.

Be aware: the use of the #shared variable within the Expression.Evaluate() function is not currently a supported feature, which means it can go away at any time.

Power BI Dataflows

Sun, 11 Nov 2018 00:00:00 +0000

A Spectacular Power BI Update

The November 2018 update of Power BI has been nothing short of spectacular. Among other things, this update introduces us to dataflows, one of the most anticpated features to come out in a while. In essence, dataflows allows us to perform complex data preparation steps within the Power BI cloud service. It’s now easier to build, standardize, and manage data preparation workflows across datasets stored on the service, all through the browser. In the coming months, dataflows will also let us leverage Azure Data Services like Azure Machine Learning for advanced analytics and AI.

Adi Regev has an excellent introduction to dataflows.

Matthew Roche provides a more in-depth look at Power BI dataflows, complete with hands-on tutorials, worked-out examples, and links to essential resources. His six-part blog series is a probably the best place to start.

For me, the ability to write Power Query M in the browser is the most exciting part of dataflows. In an upcoming post, I’ll explore how to build a series of data preparation steps in dataflows referencing Power Query M code stored somewhere else, like on a GitHub repo or a file server. I tweeted about this ability just the other day.

Post on Tony McGovern. Hacker. Artist. Storyteller. Data lover.

Handling Unsuccessful HTTP Requests in Power Query M

Map of Wisconsin midterm election results for 2018

Wisconsin Midterm Election Results for 2018

Power Query M Version Control using GitHub

Power Query M Version Control using GitHub

Expression.Evaluate() and the #shared Environment

Making it Work in Power BI Dataflows

Update 13 November 2018

Power BI Dataflows

A Spectacular Power BI Update

Expression.Evaluate() and the `#shared` Environment