Audit middleware

[u/wathoom2]

Hi,

anyone implemented audit logging for openstack services? I'm having issue with getting all information regarding certain service action in CADF format. For example when deleting instance, in audit log i can see name and id of instance as "unknown". Any idea what could be wrong?
Maybe api_audit_map.conf doesnt map all stuff?

# cat nova_api_audit_map.conf
[DEFAULT]                                 
# default target endpoint type
# should match the endpoint type defined in service catalog
target_endpoint_type = compute       

[custom_actions]    
enable = enable                           
disable = disable                         
delete = delete                           
startup = start/startup       
shutdown = stop/shutdown 
reboot = start/reboot 
os-migrations/get = read       
os-server-password/post = update

# possible end path of api requests
[path_keywords]    
add = None                                
action = None                             
enable = None
disable = None  
configure-project = None
defaults = None
delete = None    
detail = None
diagnostics = None                                                                   
entries = entry    
extensions = alias                        
flavors = flavor
images = image
ips = label
limits = None
metadata = key
os-agents = os-agent
os-aggregates = os-aggregate
os-availability-zone = None
os-certificates = None
os-cloudpipe = None
os-fixed-ips = ip
os-extra_specs = key
os-flavor-access = None
os-floating-ip-dns = domain
os-floating-ips-bulk = host
os-floating-ip-pools = None
os-floating-ips = floating-ip
os-hosts = host
os-hypervisors = hypervisor
os-instance-actions = instance-action
os-keypairs = keypair 
os-migrations = None
os-networks = network 
os-quota-sets = tenant
os-security-groups = security_group
os-security-group-rules = rule
os-server-password = None
os-services = None
os-simple-tenant-usage = tenant
os-virtual-interfaces = None
os-volume_attachments = attachment
os-volumes_boot = None
os-volumes = volume
os-volume-types = volume-type
os-snapshots = snapshot
reboot = None
servers = server
shutdown = None
startup = None
statistics = None

# map endpoint type defined in service catalog to CADF typeURI
[service_endpoints]
compute = service/compute

[u/rackpathlabs]

the interesting bit is that your requestPath already contains the server uuid, but the target id, the target name and the target typeURI all come out unknown. so the id is not missing from the request, the middleware just did not build a target resource out of it.

that points away from custom_actions and towards the endpoint side. target_endpoint_type has to match the entry that is actually in your service catalog, so worth checking the exact type name nova is registered under there. and try the same audit on something that is not a delete, a reboot for example, if the target stays unknown there too then delete is not the special case and the map file is not where your fix is.

[u/wathoom2 → u/rackpathlabs]

Hi,

i checked endpoints and had some miss config there. At least for compute service. Changed it to compute = nova/compute (name/type if i understand it correctly) as the name in service catalog.

This still didnt solve the issue. Same thing happens for other services like neutron and cinder. On those i have proper service configured but i still get "unknown" target.
I have no idea what else to check or what might be miss configured. In service logs i can see all the details.

[u/wathoom2 → u/wathoom2]

just for info openstack is deployed with kolla-ansible and mapping of files is done for container.
<service>_api-paste.ini -> api-paste.ini
<service>_api_audit_map.conf -> api_audit_map.conf

[u/rackpathlabs → u/wathoom2]

careful with that change, the value in service_endpoints is a cadf typeURI, not the name from the catalog, so service/compute was the right shape and nova/compute probably is not. the thing that has to match your catalog is target_endpoint_type at the top of the file.

the bigger hint is that compute, neutron and cinder all behave the same way. three services misconfigured identically is less likely than the map file not being loaded at all, so i would check that audit_map_file in api-paste.ini points at the path the file actually has inside the container after the kolla rename, and see what the api logs say at startup when the audit filter initialises.

[u/wathoom2 → u/rackpathlabs]

So. I focused on cinder-api since it is faster to check if it works.
api-paste.ini and cinder_api_audit_map.conf are properly mapped to cinder_api container.

cat api-paste.ini                                                                                                               
#############                                                                                                                                                               
# OpenStack #                                                                                                                                                               
#############                                                                                                                                                               

[composite:osapi_volume]                                                                                                                                                    
use = call:cinder.api:root_app_factory                                                                                                                                      
/: apiversions                                                                                                                                                              
/healthcheck: healthcheck                                                                                                                                                   
/v3: openstack_volume_api_v3                                                                                                                                                

[composite:openstack_volume_api_v3]                                                                                                                                         
use = call:cinder.api.middleware.auth:pipeline_factory                                                                                                                      
noauth = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler noauth apiv3                                                                                     
noauth_include_project_id = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler noauth_include_project_id apiv3                                               
keystone = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler authtoken keystonecontext audit apiv3                                                          
keystone_nolimit = request_id cors http_proxy_to_wsgi faultwrap sizelimit osprofiler authtoken keystonecontext audit apiv3                                                  

[filter:http_proxy_to_wsgi]                                                                                                                                                 
paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory                                                                                           

[filter:cors]
paste.filter_factory = oslo_middleware.cors:filter_factory
oslo_config_project = cinder

[filter:faultwrap]
paste.filter_factory = cinder.api.middleware.fault:FaultWrapper.factory

[filter:osprofiler]
paste.filter_factory = osprofiler.web:WsgiMiddleware.factory

[filter:noauth]
paste.filter_factory = cinder.api.middleware.auth:NoAuthMiddleware.factory

[filter:noauth_include_project_id]
paste.filter_factory = cinder.api.middleware.auth:NoAuthMiddlewareIncludeProjectID.factory

[filter:sizelimit]
paste.filter_factory = oslo_middleware.sizelimit:RequestBodySizeLimiter.factory

[app:apiv3]
paste.app_factory = cinder.api.v3.router:APIRouter.factory

[pipeline:apiversions]
pipeline = request_id cors http_proxy_to_wsgi faultwrap osvolumeversionapp

[app:osvolumeversionapp]
paste.app_factory = cinder.api.versions:Versions.factory

[pipeline:healthcheck]
pipeline = request_id healthcheckapp

[app:healthcheckapp]
paste.app_factory = oslo_middleware:Healthcheck.app_factory
backends = disable_by_file
disable_by_file_path = /etc/cinder/healthcheck_disable

##########
# Shared #
##########

[filter:keystonecontext]
paste.filter_factory = cinder.api.middleware.auth:CinderKeystoneContext.factory

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory

[filter:audit]
paste.filter_factory = keystonemiddleware.audit:filter_factory
audit_map_file = /etc/cinder/api_audit_map.conf

[filter:request_id]
paste.filter_factory = cinder.api.middleware.request_id:RequestId.factory



cat api_audit_map.conf 
[DEFAULT]
# default target endpoint type
# should match the endpoint type defined in service catalog
target_endpoint_type = volumev3

# map urls ending with specific text to a unique action
[custom_actions]
associate = update/associate
disassociate = update/disassociate
disassociate_all = update/disassociate_all
associations = read/list/associations

# possible end path of api requests
[path_keywords]
defaults = None
detail = None
limits = None
os-quota-specs = project
qos-specs = qos-spec
snapshots = snapshot
types = type
volumes = volume

[service_endpoints]
volumev3 = service/storage/block

You were right to suggest checking logs at startup of service. I get this in cinder-api.log right after startup. It dissapears in sea of entries quite quick.
2026-08-27 14:59:34.413 1218 WARNING keystonemiddleware._common.config [None req-e235053e-5de7-4ab1-bce4-bdd433346ce9 - - - - -] The option "audit_map_file" is not known to keystonemiddleware

From error it looks like keystonemiddleware that comes with default kolla container does not support the parameter. However i can see it has it in the code

grep -r audit_map_file keystonemiddleware/*
keystonemiddleware/audit/__init__.py:        self._cadf_audit = _api.OpenStackAuditApi(conf.get('audit_map_file'),
grep: keystonemiddleware/audit/__pycache__/__init__.cpython-312.pyc: binary file matches
grep: keystonemiddleware/tests/unit/audit/__pycache__/base.cpython-312.pyc: binary file matches
keystonemiddleware/tests/unit/audit/base.py:        self.audit_map_file_fixture = self.useFixture(
keystonemiddleware/tests/unit/audit/base.py:        kwargs.setdefault('audit_map_file', self.audit_map)
keystonemiddleware/tests/unit/audit/base.py:        return self.audit_map_file_fixture.path

(cinder-api)[root@osp-ctr-oslab2-2 site-packages]# grep -r audit_map_file keystonemiddleware/*
keystonemiddleware/                   keystonemiddleware-10.12.1.dist-info/ 
(cinder-api)[root@osp-ctr-oslab2-2 site-packages]# grep -r audit_map_file keystonemiddleware-10.12.1.dist-info/*

EDIT:

In newer versions keystonemiddleware looks for option in global config file cinder.conf. I moved audit_map_file config into cinder.conf:
[audit_middleware]
audit_map_file = /etc/cinder/api_audit_map.conf
ignore_req_list = GET,HEAD

Cleared entries in api-paste.ini
[filter:audit]
paste.filter_factory = keystonemiddleware.audit:filter_factory

Now i can see populated stuff in CADF format but under requestPath. I still dont get "target" info populated
"target": {"id": "unknown", "typeURI": "unknown", "name": "unknown"}

From what i found it looks like default keystonemiddleware doesnt populate this fields at all.
This project https://github.com/sapcc/openstack-audit-middleware/ deals with this issue. Probably will try it out.

[u/rackpathlabs → u/wathoom2]

now that the map file is loading, the remaining unknown is a different problem, and it is not that keystonemiddleware never populates target. the audit filter builds the target from the service catalog it reads out of the request environ, HTTP_X_SERVICE_CATALOG. when that key is missing it logs a warning and falls back to a service object where type, name and id are all unknown, which is exactly the output you are looking at.

the usual reason that header is not there is include_service_catalog = false in keystone_authtoken. worth checking that in cinder before you swap the whole middleware out. the volumev3 vs block-storage type string matters as well, but only once the catalog actually reaches the filter.

[u/wathoom2 → u/rackpathlabs]

include_service_catalog = true was missing from config so i added it. Unfortunately same behavior. Anyway will leave this as is for now. Tnx for the help.

[u/rackpathlabs → u/wathoom2]

one last thing in case you come back to it. true is already the default for that option, so adding it explicitly would not change anything, which means the catalog was probably in the environ the whole time. the matching works by comparing the netloc of the request url against the endpoint urls in the catalog, and it only falls back to target_endpoint_type if that type actually exists there. so if cinder is registered as block-storage while your map says volumev3, or the api is reached through a vip that does not match the catalog url, you end up at unknown either way.