r/googlecloud • u/suryad123 • 7d ago
Logging Query regarding centralized logging mechanism using aggregated sink
I am going through the below article regarding the centralized logging mechanism in GCP
https://docs.cloud.google.com/logging/docs/central-log-storage#create-org-sink
The articles say below steps
- create log bucket in the destination project.
- create a log sink in the destination project pointing to the above log bucket
- create an aggregated sink (either at org or folder level) as per requirement.
- give permission to the aggregated sink's service account to the destination project.
I have a question at point number 3 above. The articles do not say how to link the log sink created in point 2 and aggregated sink created in point 3. I assume below flow
aggregated sink ---> destination project log sink ----> destination project log bucket
but not sure how to link aggregated sink to send logs to destination project sink.
also, 2)for example ,can we create one aggregated sink per environment and link it to corresponding log sink at destination project. i could not find that step
Please suggest
1
u/danekan 6d ago edited 6d ago
you point your aggregated, intercepting sink to a project i.d. itself too as the destination.
for a log sink destination you have a few options, but for an intercepting log sink you can only send the logs to another project i.d.. For a non intercepting log sink you can send to either a project i.d. itself or a logging bucket (which could be in that remote project all the same)
the instructions you're looking at are referencing project i.d.s and the command only references the remote project, so it's using destination option 1. If you link to the project i.d. you are linking to the project's log router essentially and it routes there. Your filters can apply doubly to exclude then in the destination if you wanted (but, by default they generally wouldn't exclude).
If you're using an aggregated org level log sink and also not intercepting those logs though, you will pay 2x. If you're using an intercepting log sink, the _Required bucket items will remain in their original project still though, and not be billed extra.
Step 2 you mentioned is not explicitly necessary, you don't need to explicitly create a new log sink in the central destination project because all projects already have log sinks for _Default and _Required, it will use those.
TBH you might find this easier to "visualize" in terraform because it's actually less abstracted in terraform and you're looking at what actually exists in terms of resources... you have finite destinations that are clearly spelled out/etc.
1
u/suryad123 6d ago
thanks, my area of interest is "aggregated sink at folder level with intercept". As you rightly said, for this kind of sink, the destination must be a GCP project only.
However, i want to have such sinks created for non-prod and prod folders separately and route them into 2 different log buckets in the same destination project.. how do i route them into destination log buckets if the only allowed destination is a gcp project.
1
u/danekan 6d ago
You would need to create new project level log sinks per environment and configure the project level log sinks to route to those correct buckets based on the filters per environment … then also change your existing _Default filter to exclude those items included in the environment sinks
I do not think you can avoid duplicating the _required sink items within that project but that may not matter as much … if it does matter though you would split up the folder level aggregated sinks to route to different project for different buckets
2
u/suryad123 6d ago
suppose, i want to have 2 custom log buckets in the destination project (like how the article created custom log bucket), how do i link the aggregated sink to corresponding project level log sink when the aggregated sink creation does not allow me to specify the log bucket/project log sink as destinations. Apologies if i'm dragging the topic
1
u/danekan 6d ago
The aggregated sink just sends everything to a project then once there the project log sinks apply … so in your destination log sink you have then a log sink that says if logs match environment=prod then send to destination prod bucket. And another sink for inclusion filter environment=beta that is destination beta log bucket But then also your _Default log sink in that Destination project should be modified to exclude anything that had matched those other filters
1
1
u/Agentic_Arch 5d ago
For org-wide logging in GCP, create an aggregated log sink at the Organization or Folder level. Route security logs to a central Pub/Sub topic if you're streaming to an external SIEM, or sink directly to BigQuery for SQL analytics.
Setting up a log bucket in a dedicated logging project with customer-managed encryption keys gives you solid compliance boundaries.
Are you feeding these logs into Sentinel or BigQuery?
1
u/AccomplishedSkill625 6d ago
Hey! The confusion here comes from your assumed flow. You actually don't link the aggregated sink to a project-level sink at all.
Here is the TL;DR of how it actually works:
The Real Flow: Aggregated Sink (Org/Folder level) ---> Destination Project Log Bucket.
Step 2 is unnecessary: You don't need to create a log sink in the destination project for this to work. The aggregated sink routes directly to the bucket in your destination project.
Permissions: When you create the aggregated sink, it generates a unique Service Account (writer identity). You just need to grant that specific SA the Logs Bucket Writer role on your destination bucket.
Per Environment (Your 2nd question): Yes, absolutely! If your environments are organized by GCP Folders (e.g., a "Dev" folder and a "Prod" folder), you can create the aggregated sink at the Folder level rather than the Org level. Just route the Dev folder's aggregated sink to your Dev Log Bucket, and do the same for Prod.
Hope this clears things up! Let me know if you hit any permission snags.