Grype performance issue #737
Description
What happened: Hi! I used the Harbor Scanner Adapter (https://github.com/anchore/harbor-scanner-adapter) to scan images from Harbor UI.
Harbor set the Http client timeout to 5 seconds (https://github.com/goharbor/harbor/blob/main/src/pkg/scan/rest/v1/client.go#L98) and each time, Harbor ask the vulnerabilities reports, the harbor http request ends with an Context Deadline Exceeded causing by grype performance, i guess..
After debugging each stack (adapter, api, policy etc..) : i observe a long request (around 12 seconds) on sqlite dabase causing the client http timeout :
[service:policy-engine] 2022-04-28 14:03:35+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.services.policy_engine.api.controllers.synchronous_operations/query_vulnerabilities_get()] [INFO] Querying vulnerabilities
[service:policy-engine] 2022-04-28 14:03:35+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/read_lock_access()] [DEBUG] Attempting to get read access for the grype_db lock
[service:policy-engine] 2022-04-28 14:03:35+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/read_lock_access()] [DEBUG] Acquired read access for the grype_db lock
[service:policy-engine] 2022-04-28 14:03:35+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/query_vulnerabilities()] [DEBUG] Querying grype_db for vuln_id: ['CVE-2017-10118', 'CVE-2014-3637', 'CVE-2015-4042', 'CVE-2009-5155', 'CVE-2021-36085', 'CVE-2022-21434', 'CVE-2017-13745', 'CVE-2015-8982', 'CVE-2021-36086', 'CVE-2022-21476', 'CVE-2014-3635', 'CVE-2020-12399', 'CVE-2015-4041', 'CVE-2022-1271', 'CVE-2022-21496', 'CVE-2015-8983', 'CVE-2019-17450', 'CVE-2018-25032', 'CVE-2017-18248', 'CVE-2017-10176', 'CVE-2021-22946', 'CVE-2021-36084', 'CVE-2014-3639', 'CVE-2022-21426', 'CVE-2020-3898', 'CVE-2020-35521', 'CVE-2014-9939', 'CVE-2014-3638', 'CVE-2017-14132', 'CVE-2017-13752', 'CVE-2021-36087', 'CVE-2022-21443', 'CVE-2016-9397', 'CVE-2015-2808', 'CVE-2015-8985', 'CVE-2015-8984'], namespace: ['rhel:7', 'nvdv2:cves'], affected_package: None
[service:policy-engine] 2022-04-28 14:03:35+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/grype_session_scope()] [DEBUG] Opening grype_db session: <sqlalchemy.orm.session.Session object at 0x7f24f39c8340>
[service:policy-engine] 2022-04-28 14:03:35+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/query_vulnerabilities()] [DEBUG] grype_db sql query for vulnerabilities lookup: SELECT vulnerability.pk AS vulnerability_pk, vulnerability.id AS vulnerability_id, vulnerability.package_name AS vulnerability_package_name, vulnerability.namespace AS vulnerability_namespace, vulnerability.version_constraint AS vulnerability_version_constraint, vulnerability.version_format AS vulnerability_version_format, vulnerability.cpes AS vulnerability_cpes, vulnerability.related_vulnerabilities AS vulnerability_related_vulnerabilities, vulnerability.fixed_in_versions AS vulnerability_fixed_in_versions, vulnerability.fix_state AS vulnerability_fix_state, vulnerability.advisories AS vulnerability_advisories, vulnerability_metadata.id AS vulnerability_metadata_id, vulnerability_metadata.namespace AS vulnerability_metadata_namespace, vulnerability_metadata.data_source AS vulnerability_metadata_data_source, vulnerability_metadata.record_source AS vulnerability_metadata_record_source, vulnerability_metadata.severity AS vulnerability_metadata_severity, vulnerability_metadata.urls AS vulnerability_metadata_urls, vulnerability_metadata.description AS vulnerability_metadata_description, vulnerability_metadata.cvss AS vulnerability_metadata_cvss
[service:policy-engine] FROM vulnerability_metadata LEFT OUTER JOIN vulnerability ON vulnerability.id = vulnerability_metadata.id AND vulnerability.namespace = vulnerability_metadata.namespace
[service:policy-engine] WHERE vulnerability.id IN ([POSTCOMPILE_id_1]) AND vulnerability.namespace IN ([POSTCOMPILE_namespace_1])
[service:policy-engine] 2022-04-28 14:03:39+0000 [-] "10.128.72.2" - - [28/Apr/2022:14:03:38 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.22"
[service:policy-engine] 2022-04-28 14:03:39+0000 [-] "10.128.72.2" - - [28/Apr/2022:14:03:38 +0000] "GET /health HTTP/1.1" 200 5 "-" "kube-probe/1.22"
[service:policy-engine] 2022-04-28 14:03:47+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/grype_session_scope()] [DEBUG] Closing grype_db session: <sqlalchemy.orm.session.Session object at 0x7f24f39c8340>
[service:policy-engine] 2022-04-28 14:03:47+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/read_lock_access()] [DEBUG] Releasing read access for the grype_db lock
[service:policy-engine] 2022-04-28 14:03:47+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/read_lock_access()] [DEBUG] Attempting to get read access for the grype_db lock
[service:policy-engine] 2022-04-28 14:03:47+0000 [_GenericHTTPChannelProtocol,11,10.128.80.15] [PoolThread-twisted.internet.reactor-1] [anchore_engine.clients.grype_wrapper/read_lock_access()] [DEBUG] Acquired read access for the grype_db lock
What you expected to happen: Sometimes, i observe the same request with a better response time, around 1 seconds.
I don't found any documentation to tune (memory, connection, cpu etc...) the sqlite of grype.
So i expect anchore response time under 5 seconds to fit the anchore integration with harbor.
How to reproduce it (as minimally and precisely as possible): Anchore is deployed in version 1.1.0 on Openshift with helm-chart https://github.com/anchore/anchore-charts
Anything else we need to know?: Nothing
Environment:
- Output of
grype version: - OS (e.g:
cat /etc/os-releaseor similar):
cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="49.84.202203081945-0"
Metadata
Metadata
Assignees
Labels
Type
Projects
Status