[add_kubernetes_metadata][Metricset state_namespace] Fix namespace filter - #39934

Proposed commit message

This PR:

1. Adds namespace filter to add_kubernetes_metadata watchers.
2. At metricset level, it fixes the state_namespace to also consider namespace filter like all the other namespaced resources.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Relates to #38978.

How to test locally

Learn how to run your own filebeat here. In next section, I explain my tests.

Results

I have a cluster with two namespaces: default and kube-system.

    filebeat.inputs:
      - type: filestream
        id: kubernetes-container-logs
        paths:
          - /var/log/containers/*.log
        parsers:
          - container: ~
        prospector:
          scanner:
            fingerprint.enabled: true
            symlinks: true
        file_identity.fingerprint: ~
        processors:
          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              namespace: default
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"

The documents fields annotated with data coming from the processor need to be coming from pods inside default namespace. Checking in discover I can see:

And:

So only pods from default get annotated as expected.

If I now remove the namespace from the processor and leave it like this:

          - add_kubernetes_metadata:
              host: ${NODE_NAME}
              matchers:
                - logs_path:
                    logs_path: "/var/log/containers/"

I can see all documents get annotated with metadata: