Federating REST API sources

Access for free via a friend’s link

Recently, we jumped into creating our own data sources to enable custom advanced REST API applications.

However, since version 16.2 of databricks runtime, we have another way of connecting to REST API sources. We can define a connection to the HTTP source in Unity Catalog. For the purpose of this article, we will use the publicly available REST API schema available at https://jsonplaceholder.typicode.com/.

We can also do that programmatically:

CREATE CONNECTION jsonplaceholder
  TYPE HTTP
  OPTIONS (
    host 'https://jsonplaceholder.typicode.com',
    port '443',
    base_path '/',
    bearer_token 'test'
  );

We can not (at least for now) define catalog in Unity, but we can use that connection inside SQL and probably soon in more places as there is some info about ingestion pipelines in error.

After the registration source, we can use the new function http_request() to connect to REST API data sources.

http_request( 
        conn => "jsonplaceholder",
        method => "GET",
        path => "/posts/1",
        params=> map(),
        json => '',
        headers => map() )

As we see in the example below, it nicely complements the new VARIANT type, which transforms JSON into a highly optimized Object.

Hubert Dudek (author)

If you like this blog post, consider buying me a coffee :-) https://ko-fi.com/hubertdudek