How The CARTO Platform Enables The Creation Of Advanced Data Visualizations With Google Maps Platform And Deck.gl

Editor’s note: Today’s blog post is from Alberto Asuero, CTO of CARTO,  the location intelligence platform. Today he shares more details about the source of the data for advanced data visualizations created with Google Maps Platform and deck.gl and how the CARTO platform enables this workflow.

During Google Cloud Next in October, the Google team announced the newest release of the deck.gl visualization library, thanks to a collaboration with our geospatial company CARTO and the vis.gl Technical Steering Committee (TSC). The deck.gl release includes a deep integration with the new WebGL-powered features in the Maps JavaScript API that allows deck.gl to render 2D and 3D visualizations directly on the Google basemap.

Partner with aster.cloud
for your next big idea.
Let us know here.

Our team built an example app that visualizes a variety of data sources that show the potential for electrification of truck fleets in Texas. This app showcases the different types of advanced data visualizations that can be created with Google Maps Platform and deck.gl. Today, I want to share more details about the source of the data for these visualizations and how the CARTO platform enables this workflow.

Google Cloud provides a strong serverless data warehousing solution, BigQuery, with support for geospatial queries. When you are dealing with spatial data, creating maps to explore and visualize these datasets is an important and common need. The CARTO Spatial Extension for BigQuery provides an easy way to create connections to the data warehouse, design a map with data coming from BigQuery tables, and then add these visualizations to a web app using deck.gl.

An example of retrieving a Map ID from CARTO Builder
Different custom styles can be applied to the map directly in CARTO Builder

 

Making a simple map

To create a simple map using the CARTO platform, you can sign up for a trial account. Once you have signed in, you can set up a connection to your BigQuery instance using a service account. Then, you can go to the Data Explorer and browse the available datasets to find the table you want to use as the datasource in your map. For more information, check out the CARTO documentation.

Read More  Automating Log Uploads With Gcloud Transfer

 

CARTO  Builder’s Data Explorer allows you to preview different geospatial datasets

 

To create a visualization of power transmission lines in Texas, you can start with the Texas state boundary to provide some context. In the Data Explorer, you can preview the table and click the “Create map” button in the top-right corner to start designing your visualization.

Using the CARTO Builder map making tool, select one of the available Google vector basemap styles and customize the layer style.

The result of an executed geospatial query displayed in CARTO Builder

 

You can visualize tables and the results from queries executed in the data warehouse, which is a powerful feature because you can also execute spatial analysis functions using SQL, including those from the CARTO Analytics Toolbox. In this case, you can intersect the lines in the table containing all the U.S. transmission lines within the Texas boundary. Click on the “Add source from…” button and select the “Custom Query (SQL)” option to add the following query:

SELECT * 
FROM cartobq.nexus_demo.transmission_lines
WHERE ST_INTERSECTS(
  geometry, 
  (SELECT geom FROM cartobq.nexus_demo.texas_boundary_simplified)
);
An example of a geospatial query being executed in CARTO Builder

 

Click the “Run” button, and the query is executed in BigQuery. The results are sent back to the Builder tool. Perform some style customizations in the new layer, and your map is ready.

Results are automatically visualized when they are returned from BigQuery

 

Before adding the map to the Google Maps Platform application, you’ll need to make it public. Click on the “Share” button and select the “Developers” tab to copy the map ID.

Generate a Map ID for use with Google Maps Platform web and mobile SDKs from the share menu in CARTO Building

 

Now, you can add the visualization into your Google Maps Platform application, which is as easy as adding these four lines of code:

 

const cartoMapId = 'b502bf53-877d-4e89-b5ad-71982cac431d';
deck.carto.fetchMap({cartoMapId}).then(({layers}) => {
  const overlay = new deck.GoogleMapsOverlay({layers});
  overlay.setMap(map);
});

 

You can use the map ID copied from CARTO Builder to call the fetchMap function. This function connects to the platform and retrieves all the information needed for the visualization, including a collection of deck.gl layers with all the styling properties you’ve specified. Create an instance of the deck.gl  GoogleMapsOverlay with this collection of layers and add it to the map.

Read More  Intel To Collaborate With Microsoft On DARPA Program

You can see the full example in this fiddle.

Full example available in JSFiddle

 

Visualizing very large datasets

One of the main features of BigQuery is the ability to scale processing to massive datasets. With the CARTO platform, you can also visualize very large datasets using tilesets, an optimized data structure containing pre-generated vector tiles for fast visualization. Tilesets are generated within BigQuery using the Analytics Toolbox functions in a parallelized process that can handle billions of points.

For example, you can create a visualization using tilesets with the whole dataset of transmission lines for the U.S., more than 100MB of geometries.

The issue with these large datasets is that they do not fit in memory all at once, so you need to split them into tiles for them to be rendered progressively. CARTO takes care of this, allowing you to create tilesets directly in BigQuery or dynamically generate them on the fly.

A tileset generated in BigQuery and displayed in CARTO Builder

 

This method for data loading in maps can scale as much as needed; for example, take a look at this 17 billion point visualization of vessel data.

Dataset of 17 billion points, rendered using a custom tileset

 

What about live data?

BigQuery supports streaming data that is continuously updated. In these scenarios, you want to be able to update your visualization at regular intervals, as the data changes. It’s easy to update this visualization using deck.gl. You just need to set the autoRefresh parameter to true when fetching the map and specify the function you want to execute when new data is downloaded:

const {layers} = await deck.carto.fetchMap({
  cartoMapId,
  autoRefresh: true,
  onNewData: (parsedMap) => { … }
});

 

You can add points to a table with an INSERT function on the BigQuery console and see the data updated on the map in real time.

Datasets in BigQuery can be updated on the fly on visualized with CARTO

 

Going further

In addition to the simple ways to create visualizations shown above, deck.gl has the flexibility to create a wide variety of visualizations. The CARTO platform provides you with the functionality to access data from your data warehouse and create these data visualizations with advanced cartographic capabilities, but you can extend it and go beyond that using any of the advanced visualizations available in the deck.gl layer catalog.

Read More  Security Command Center - Increasing Operational Efficiency With New Mute Findings Capability

There are two additional options that give you more control over the deck.gl code. The first one is to use the CartoLayer directly without fetchMap. You’ll need to indicate the connection to use from the CARTO platform and the data source type and name or query. Then we can specify the styling properties.

 

const overlay = new deck.GoogleMapsOverlay({
  layers: [
    new deck.carto.CartoLayer({
      connection: 'bqconn',
      type: deck.carto.MAP_TYPES.TABLE,
      data: `cartobq.public_account.retail_stores`,
      getFillColor: [238, 77, 90],
      pointRadiusMinPixels: 6,
    }),
  ],
});

 

The second option is to use the fetchLayerData function that allows you to have more control over the format used for data transfer between BigQuery and your application and can be used with advanced visualizations that require an specific data format like ArcLayer, H3HexagonLayer or TripsLayer.

 

deck.carto.fetchLayerData({
  type: deck.carto.MAP_TYPES.TABLE,
  source: `cartobq.geo_for_good_meetup.texas_pop_h3`,
  connection: 'bqconn',
  format: deck.carto.FORMATS.JSON,
  credentials: {
     accessToken: 'eyJhbGciOiJIUzI1NiJ9.eyJhIjoiYWNfbHFlM3p3Z3UiLCJqdGkiOiI1YjI0OWE2ZCJ9.Y7zB30NJFzq5fPv8W5nkoH5lPXFWQP0uywDtqUg8y8c'
   }
 }).then(({data}) => {
  const layers= [
    new deck.H3HexagonLayer({
      id: 'h3-hexagon-layer',
      data,
      extruded: true,
      getHexagon: d => d.h3,
      getFillColor: [182, 0, 119, 150],
      getElevation: d => d.pop,
      elevationScale: 2.5,
      parameters: {
        blendFunc: [luma.GL.SRC_ALPHA, luma.GL.DST_ALPHA],
        blendEquation: luma.GL.FUNC_ADD
      }
    })
  ];
  const overlay = new deck.GoogleMapsOverlay({layers});
  overlay.setMap(map);
 });

 

For complete code using both options, take a look at these examples.

Example of using the deck.gl Hexagon Layer visualization with Google Maps Platform and CARTO

 

Learn more

You can access demos and documentation on the deck.gl docs website and the CARTO Documentation Center. If you have questions, you can ping the CARTO team on the CARTO Users Slack workspace.

For more information on Google Maps Platform, visit the Google Maps Platform website.

By: Alberto Asuero (CTO of CARTO)

Source: Google Cloud Blog

For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!
Our humans need coffee too! Your support is highly appreciated, thank you!

Related Topics
You May Also Like