aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
aster.cloud aster.cloud
  • /
  • Platforms
    • Public Cloud
    • On-Premise
    • Hybrid Cloud
    • Data
  • Architecture
    • Design
    • Solutions
    • Enterprise
  • Engineering
    • Automation
    • Software Engineering
    • Project Management
    • DevOps
  • Programming
    • Learning
  • Tools
  • About
  • DevOps
  • Software Engineering

More Support For Structured Logs In New Version Of Go Logging Library

  • aster.cloud
  • July 17, 2022
  • 5 minute read

The new version of the Google logging client library for Go has been released. Version 1.5 adds new features and bug fixes including new structured logging capabilities that complete last year’s effort to enrich structured logging support in Google logging client libraries.

Here are few of the new features in v1.5:


Partner with aster.cloud
for your next big idea.
Let us know here.



From our partners:

CITI.IO :: Business. Institutions. Society. Global Political Economy.
CYBERPOGO.COM :: For the Arts, Sciences, and Technology.
DADAHACKS.COM :: Parenting For The Rest Of Us.
ZEDISTA.COM :: Entertainment. Sports. Culture. Escape.
TAKUMAKU.COM :: For The Hearth And Home.
ASTER.CLOUD :: From The Cloud And Beyond.
LIWAIWAI.COM :: Intelligence, Inside and Outside.
GLOBALCLOUDPLATFORMS.COM :: For The World's Computing Needs.
FIREGULAMAN.COM :: For The Fire In The Belly Of The Coder.
ASTERCASTER.COM :: Supra Astra. Beyond The Stars.
BARTDAY.COM :: Prosperity For Everyone.

 

  • Faster and more robust way to detect and capture Google Cloud resources that the application is running on.
  • Automatic source location detection to support log observability for debugging and troubleshooting.
  • W3C header traceparent for capturing tracing information within the logged entries.
  • Better control over batched ingestion of the log entries by supporting the partialSuccess flag within Logger instances.
  • Support for out-of-process ingestion with redirection of the logs to stdout and stderr using a structured logging format.

 

Let’s look into each closer:

Resource detection

Resource detection is an existing feature of the logging library. It detects a resource on which an application is running. Retrieves the resource’s metadata. And implicitly adds this metadata to each log entry the application ingests using the library. It is especially useful for applications that run on Google Cloud since it collects a lot of resource’s attributes from the Metadata server of the resource. These attributes enrich ingested logs with additional information such as a location of the VM, a name of the container or a service Id of the AppEngine service. The below Json shows a sample of the retrieved information after detecting the resource as a GKE container and retrieving resource metadata according to the documentation.

 

{
  "type": "k8s_container",
  "labels": {
    "project_id": "dev-env-060122",
    "location": "us-central1-a",
    "cluster_name": "dev-test-cluster-47fg",
    "namespace_name": "default",
    "pod_name" : "frontend-4fgd4",
    "container_name": "frontend-4fgd4-acgf12a5"
  }
}

 

The implementation is optimized to avoid performance degradation during the data collection process. Previously, the heuristic for identifying the resource was heavily based on environment variables which could result in many false positives. Additionally, the implementation performed too many queries to the metadata server which could sometimes cause delayed responses. In the 1.5 release the heuristic was updated to use additional artifacts beside the environment variables in the resource detection logic and the number of the queries to the metadata server was reduced to a bare minimum. As a result, false detection of GCP resources is decreased by an order of magnitude and the performance penalties to run the heuristic in non-GCP resources is decreased as well. The change does not affect the ingestion process and does not require any changes in the application’s code.

Read More  Google Cloud Next 2019 | How Customers Are Migrating Hadoop to Google Cloud Platform

Source location capturing

It is useful to capture the location in code where the log was ingested. While the main usage is in troubleshooting and debugging it can be useful in other circumstances. In this version of the library you can configure your logger instance to capture the source location metadata for each log entry ingested using Logger.Log() or Logger.LogSync() functions. Just pass the output of the SourceLocationPopulation() as a LoggerOption argument in the call to Client.Logger() when creating a new instance of the logger. The following snippet creates a logger instance that adds source location metadata into each ingested log with severity set to Debug:

 

logger := client.Logger("debug-logger",
           logging.SourceLocationPopulation(PopulateSourceLocationForDebugEntries))

 

The function SourceLocationPopulation() accepts the following constants:

  • logging.DoNotPopulateSourceLocation ‒ is a default configuration that prevents capturing the source location in the ingested logs
  • logging.PopulateSourceLocationForDebugEntries ‒ adds the source location metadata into logs with Debug severity.
  • logging.AlwaysPopulateSourceLocation ‒ populates the source location in all ingested logs.

This feature has to be enabled explicitly because the operation of capturing the source location in Go may increase the total execution time of the log ingestion by a factor of 2. It is strongly discouraged to enable it for all ingested logs.

Use W3C context header for tracing

You could add tracing information with your logs in the previous versions of the library. The way to do it was directly, by providing trace and span identification and, optionally, the sampling flag. The following code demonstrates the manual setting of the trace and span identifiers:

 

logger := client.Logger("my-log")
// …
logger.Log(
  logging.Entry{
    Payload: "keep tracing",
    Trace: "4bf92f3577b34da6a3ce929d0e0e4736",
    SpanID: "00f067aa0ba902b7",
  })

 

Read More  How Let’s Enhance Uses NVIDIA AI And GKE To Power AI-Based Photo Editing

Or indirectly, by passing an instance of the http.Request as a part of the Http request metadata:

 

logger := client.Logger("my-log")
// …
func MyHandler(w http.ResponseWriter, r *http.Request) {
  logger.log(
    logging.Entry{
      Payload: "My handler invoked",
      HttpRequest: &logging.HttpRequest{
        Request: r,
      },
    })
}

 

In the latter case, the library will try to pull tracing information from the x-cloud-tracing-context header. From this release, the library also supports W3C tracing context header. If both headers are present, the tracing information is captured from the W3C traceparent header.

Out-of-process logs’ ingestion

By default the library supports synchronous and asynchronous log ingestions by calling the Cloud Logging API directly. In certain cases the log ingestion is better to be done using external logging agents or built-in support for logs collection. In this release, you can configure a logger instance to write logs to stdout or stderr instead of ingesting it to Cloud Logging directly. The following example creates a logger that redirects logs to stdout using specially formatted Json string:

 

logger := client.Logger("not-ingesting-log", RedirectAsJSON(os.Stdout)
logger.Log(logging.Entry{Severity: logging.Debug, Payload: "out of process log"})

 

The above code will print something like the following line to the standard output:

 

{"message":"out of process log", "severity":"DEBUG", "timestamp":"seconds:1656381253"}

 

In some circumstances, when the standard output cannot be used for printing logs, the logger can be configured to redirect output to the standard error (os.Stderr) with the same effect.

There are a couple of things to be aware of when you use the out-of-process logging:

  • Methods Logger.Log() and Logger.LogSync() behave the same way when the logger is configured with the out-of-process logging option. They write the Jsonified logs to the provided io.Write writer. And an external logging agent determines the logs’ collection and ingestion.
  • You do not have control over the Log ID. All logs that are ingested by the logging agent or the built-in support of the managed service (e.g. Cloud Run) will use the Log ID that is determined out-of-process.
Read More  Learn How To Tackle Supply Chain Disruptions With SAP IBP And Google Cloud

More control over batch ingestion

When you ingest logs using Logger.Log() function, the asynchronous ingestion batches multiple log entries together and ingest them using the entries.write Logging API. If the ingestion of any of the aggregate logs fails, no logs get ingested. Starting with this release you can control this logic by opting in the partial success flag. When the flag is set, the Logging API tries to ingest all logs, even if some other log entry fails due to a permanent error such as INVALID_ARGUMENT or PERMISSION_DENIED. This option can be opted-in when creating a new logger using the PartialSuccess logger option:

 

logger := client.Logger("my-log", PartialSuccess())

 

Wrapping up

When you upgrade to version 1.5 you get a more robust and deterministic resource detection algorithm while keeping the behavior of the library unchanged. Additional functionality such as out-of-process ingestion, source location or batch ingestion control can be opted-in using the logger options. With these new features and fixes the behavior of the library becomes more deterministic and robust.

Learn more about the release at go.pkg.dev. Please also visit the library’s project on Github.

 

 

By: Leonid Yankulin (Developer Relations Engineer)
Source: Google Cloud Blog


For enquiries, product placements, sponsorships, and collaborations, connect with us at [email protected]. We'd love to hear from you!

Our humans need coffee too! Your support is highly appreciated, thank you!

aster.cloud

Related Topics
  • devops
  • Go
  • Go Language
  • Google Cloud
  • Tutorials
You May Also Like
aster-cloud-erp-bill_of_materials_2
View Post
  • Software
  • Software Engineering

What is an SBOM (software bill of materials)?

  • July 2, 2025
View Post
  • Software Engineering
  • Technology

Claude 3.7 Sonnet and Claude Code

  • February 25, 2025
View Post
  • Engineering
  • Software Engineering

This Month in Julia World

  • January 17, 2025
View Post
  • Engineering
  • Software Engineering

Google Summer of Code 2025 is here!

  • January 17, 2025
View Post
  • Software Engineering

5 Books Every Beginner Programmer Should Read

  • July 25, 2024
View Post
  • DevOps
  • Engineering
  • Platforms

How To Fail At Platform Engineering

  • March 11, 2024
Ruby
View Post
  • Software Engineering

How To Get Started With A Ruby On Rails Project – A Developer’s Guide

  • January 27, 2024
View Post
  • Engineering
  • Software Engineering

5 Ways Platform Engineers Can Help Developers Create Winning APIs

  • January 25, 2024

Stay Connected!
LATEST
  • 1
    Building secure, scalable AI in the cloud with Microsoft Azure
    • July 5, 2025
  • 2
    Turns out OpenAI is the customer behind Oracle’s mysterious $30 billion cloud deal
    • July 3, 2025
  • aster-cloud-erp-bill_of_materials_2 3
    What is an SBOM (software bill of materials)?
    • July 2, 2025
  • aster-cloud-sms-pexels-tim-samuel-6697306 4
    Send SMS texts with Amazon’s SNS simple notification service
    • July 1, 2025
  • Camping 5
    The Summer Adventures : Camping Essentials
    • June 27, 2025
  • aster-cloud-website-pexels-goumbik-574069 6
    Host a static website on AWS with Amazon S3 and Route 53
    • June 27, 2025
  • 7
    A looming hyperscaler exodus? UK IT leaders are thinking of ditching US cloud providers – here’s why
    • June 26, 2025
  • Prioritize security from the edge to the cloud
    • June 25, 2025
  • 6 edge monitoring best practices in the cloud
    • June 25, 2025
  • Genome 10
    AlphaGenome: AI for better understanding the genome
    • June 25, 2025
about
Hello World!

We are aster.cloud. We’re created by programmers for programmers.

Our site aims to provide guides, programming tips, reviews, and interesting materials for tech people and those who want to learn in general.

We would like to hear from you.

If you have any feedback, enquiries, or sponsorship request, kindly reach out to us at:

[email protected]
Most Popular
  • 1
    There’s a ‘cloud reset’ underway, and VMware Cloud Foundation 9.0 is a chance for Broadcom to pounce on it
    • June 17, 2025
  • 2
    ‘This was a very purposeful strategy’: Pure Storage unveils Enterprise Data Cloud in bid to unify data storage, management
    • June 18, 2025
  • 3
    Pure Accelerate 2025: All the news and updates live from Las Vegas
    • June 18, 2025
  • Oracle adds xAI Grok models to OCI
    • June 17, 2025
  • What is cloud bursting?
    • June 18, 2025
  • /
  • Technology
  • Tools
  • About
  • Contact Us

Input your search keywords and press Enter.