Professional Documents
Culture Documents
html
Claircore
Claircore is the engine behind the Clair v4 container security solution. The Claircore package
exports our domain models, interfaces necessary to plug into our business logic, and a
default set of implementations. This default set of implementations define our support
matrix and consists of the following distributions and languages:
• Ubuntu
• Debian
• RHEL
• Red Hat Container First content
• SUSE
• Oracle
• Alpine
• AWS Linux
• VMWare Photon
• Python
• Java
• Go
• Ruby
Claircore relies on PostgreSQL for its persistence and the library will handle migrations if
configured to do so.
Indexer
Matcher
When a claircore.Manifest is submitted to Libindex, the library will index its constituent parts
and create a report with its findings.
1 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
2 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Getting Started
The following documentation helps a beginner learn to use Claircore.
• Libindex Usage
• Libvuln Usage
3 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Libindex Usage
Libindex is the Go package responsible for fetching container image layers, identifying
packages, distributions, and repositories within these layers, and computing a final
coalesced Index Report.
Usage
Libindex is runtime constructed via the libindex.New method. New requires an
libindex.Opts struct.
Options
4 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
5 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Providing a nil "Ecosystems" slice will supply the default set, instructing Libindex to index for
all supported content in a layer, and is typically desired.
Construction
opts := new(libindex.Options)
// Populate with desired settings...
lib, err := libindex.New(ctx, opts, http.DefaultClient)
if err != nil {
panic(err)
}
defer lib.Close(ctx) // Remember to cleanup when done.
The constructing code should provide a valid Context tied to some lifetime.
Indexing
Indexing is the process of submitting a manifest to Libindex, fetching the manifest's layers,
indexing their contents, and coalescing a final Index Report.
Coalescing is the act of computing a final set of contents (packages, distributions, repos)
from a set of layers. Since layers maybe shared between many manifests, the final contents
of a manifest must be computed.
To perform an Index you must provide a claircore.Manifest data struture to the Index
method. The Manifest data structure describes an image manifest's layers and where they
can be fetched from.
m := new(claircore.Manifest)
// Populate somehow ...
ir, err := lib.Index(ctx, m)
if err != nil {
panic(err)
}
6 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
The Index method will block until an claircore.IndexReport is returned. The context should
be bound to some valid lifetime such as a request.
As the Indexer works on the manifest it will update its database throughout the process. You
may view the status of an index report via the "IndexReport" method.
Libindex performs its work incrementally and saves state as it goes along. If Libindex
encounters an intermittent error during the index (for example, due to network failure while
fetching a layer), when the manifest is resubmitted only the layers not yet indexed will be
fetched and processed.
State
Libindex treats layers as content addressable. Once a layer identified by a particular hash is
indexed its contents are definitively known. A request to re-index a known layer results in
returning the previous successful response.
This comes in handy when dealing with base layers. The Ubuntu base layer is seen very
often across container registries. Treating this layer as content addressable precludes the
need to fetch and index the layer every time Libindex encounters it in a manifest.
There are times where re-indexing the same layer is necessary however. At the point where
Libindex realizes a new version of a component has not indexed a layer being submitted it
will perform the indexing operation.
A client must notice that Libindex has updated one of its components and subsequently
resubmit Manifests. The State endpoint is implemented for this reason.
Clients may query the State endpoint to receive an opaque string acting as a cookie,
identifying a unique state of Libindex. When a client sees this cookie change it should re-
submit manifests to Libindex to obtain a new index report.
7 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
AffectedManifests
Libindex is capable of providing a client with all manifests affected by a set of vulnerabilities.
This functionality is designed for use with a notification mechanism.
The slice of vulnerabilities returned for each manifest hash will be sorted by
claircore.NormalizedSeverity in "most severe" descending order.
8 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Libvuln Usage
Libvuln is the Go package responsible for keeping the database of vulnerabilities consistent,
matching container image contents with vulnerabilities, and reporting diffs between updates
of the same security database.
Usage
Libvuln is runtime constructed via the libvuln.New method. New requires a
libvuln.Opts struct.
Options
9 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
10 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
// "suse"
// "ubuntu"
MatcherNames []string
Construction
11 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
opts := new(libvuln.Options)
// Populate with desired settings...
lib, err := libvuln.New(ctx, opts)
if err != nil {
panic(err)
}
defer lib.Close(ctx)
The constructing code should provide a valid Context tied to some lifetime.
On construction, New will block until the security databases are initialized. Expect some
delay before this method returns.
Scanning
m := new(claircore.Manifest)
// Populate somehow ...
ir, err := indexer.Index(ctx, m)
if err != nil {
panic(err)
}
vr, err := lib.Scan(ctx, ir)
if err != nil {
panic(err)
}
Updates API
By default, Libvuln manages a set of long running updaters responsible for periodically
fetching and loading new advisory contents into its database. The Updates API allows a client
to view and manipulate aspects of the update operations that updaters perform.
In this getting started guide, we will only cover the two methods most interesting to new
users.
12 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
UpdateOperations
This API provides a list of recent update operations performed by implemented updaters.
The UpdateOperation slice returned will be sorted by latest timestamp descending.
UpdateDiff
Mostly used by the Clair v4 notification subsystem, this endpoint will provide the caller with
any removed or added vulnerabilities between two update operations. Typically a diff takes
places against two versions of the same data source. This is useful to inform downstream
applications what new vulnerabilities have entered the system.
13 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Concepts
The following sections give a conceptual overview of how Claircore works internally.
• Vulnerability Matching
• Indexer Architecture
• Matching Architecture
• Severity Mapping
14 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Vulnerability Matching
The following describes a successful scan.
1. Updaters have ran either in the background on an interval or an offline loader has
been ran.
2. A Manifest is provided to Libindex. Libindex fetches all the layers, runs all scanner
types on each layer, persists all artifacts found in each layer, and computes an
IndexReport.
3. A IndexReport is provided to Libvuln.
4. Libvuln creates a stream of IndexRecord structs from the IndexReport and
concurrently streams these structs to each configured Matcher.
5. Libvuln computes a VulnerabilityReport aggregating all vulnerabilities discovered by all
configured Matcher implementations.
6. Sometime later the security advisory database is updated and a new request to Libvuln
will present updated vulnerability data.
15 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Indexer
claircore/indexer
The Indexer package performs Libindex's heavy lifting. It is responsible for retreiving
Manifest layers, parsing the contents of each layer, and computing an IndexReport.
To perform this action in incremental steps the Indexer is implemented as a finite state
machine. At each state transition the Indexer persists an updated IndexReport to its
datastore.
States
The following diagram expresses the possible states of the Indexer:
16 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
CheckManifest
Unindexed
FetchLayers
ScanLayers
Indexed Coalesce
IndexManifest
IndexFinished
Data Model
The Indexer data model focuses on content addressable hashes as primary keys, the
deduplication of package/distribution/repostitory information, and the recording of scan
artifacts. Scan artifacts are unique artifacts found within a layer which point to a
deduplicated general package/distribution/repository record.
17 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
ManifestLayer
Manifest
ScannedManifest
Layer
ScannedLayer
Scanner
ManifestIndex
18 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
TYPE
TYPE_ScanArtifact
Note that TYPE stands in for each of the Indexer types (i.e. Package , Repository , etc.).
HTTP Resources
Indexers as currently built may make network requests. This is an outstanding issue. The
following are the URLs used.
• https://search.maven.org/solrsearch/select
• https://catalog.redhat.com/api/containers/
• https://access.redhat.com/security/data/metrics/repository-to-cpe.json
• https://access.redhat.com/security/data/metrics/container-name-repos-map.json
19 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Matcher Architecture
claircore/internal/matcher claircore/libvuln/driver
Libvuln.Scan
Matcher
Decide vulnerability
Deicide vulnerability
When Libvuln's Scan method is called with an IndexReport it will begin the process of
matching container contents with vulnerabilities.
Each configured Matcher will be instantiated concurrently. Depending on the interfaces the
Matcher implements, one of the possible data flows will occur in the diagram above.
The provided IndexReport will be unpacked into a stream of IndexRecord structs. Each
Matcher will evaluate each record in the stream and determine if the IndexRecord is
vulnerable to a security advisory in their responsible databases.
20 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Once each Matcher returns the set of vulnerabities, Libvuln will merge the results into a
VulnerabilityReport and return this to the client.
HTTP Resources
NOTE: Remote matchers are being considered for removal.
"Remote matchers" may make HTTP requests during the matcher flow. These requests are
time-bound and errors are not reported. The following are the URLs used.
21 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Severity Mapping
Claircore will normalize a security databases's severity string to a set of defined values.
Clients may use the NormalizedSeverity field on a claircore.Vulnerability to react to
vulnerability severities without needing to know each security database's severity strings. All
strings used in the mapping tables are identical to the strings found within the relevant
security database.
• Unknown
• Negligible
• Low
• Medium
• High
• Critical
Alpine Mapping
The Alpine SecDB database does not provide severity information. All vulnerability severities
will be Unknown.
AWS Mapping
The AWS UpdateInfo database provides severity information.
22 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Debian Mapping
The Debian security tracker data provides severity information.
Oracle Mapping
The Oracle OVAL database provides severity information.
RHEL Mapping
The RHEL OVAL database provides severity information.
23 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
SUSE Mapping
The SUSE OVAL database provides severity information.
Ubuntu Mapping
The Ubuntu OVAL database provides severity information.
Photon Mapping
24 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
OSV Mapping
OSV provides severity information via CVSS vectors, when applicable. These are normalized
according to the NVD qualitative rating scale. If both v3 and v2 vectors are present, v3 is
preferred.
CVSSv3
CVSSv2
25 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
HTTP Resources
The following are the HTTP hosts and paths that Clair will attempt to talk to in a default
configuration. This list is non-exhaustive, as some servers will issue redirects and some
request URLs are constructed dynamically.
• https://secdb.alpinelinux.org/
• http://repo.us-west-2.amazonaws.com/2018.03/updates/x86_64/mirror.list
• https://cdn.amazonlinux.com/2/core/latest/x86_64/mirror.list
• https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list
• https://deb.debian.org/
• https://security-tracker.debian.org/tracker/data/json
• https://nvd.nist.gov/feeds/json/cve/1.1/
• https://linux.oracle.com/security/oval/com.oracle.elsa-*.xml.bz2
• https://packages.vmware.com/photon/photon_oval_definitions/
• https://access.redhat.com/security/data/metrics/cvemap.xml
• https://access.redhat.com/security/cve/
• https://access.redhat.com/security/data/oval/v2/PULP_MANIFEST
• https://support.novell.com/security/oval/
• https://api.launchpad.net/1.0/
• https://security-metadata.canonical.com/oval/com.ubuntu.*.cve.oval.xml
• https://osv-vulnerabilities.storage.googleapis.com/
26 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
How Tos
The following sections provide instructions on accomplish specific goals in Claircore.
27 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
The claircore team is always open to adding more distributions and languages to the library.
These databases are maintained by the distribution or language developers and reflect up-
to-date CVE and advisory data for their packages.
Implementing an Updater
The first step to adding your distribution or language to claircore is getting your security
tracker's data ingested by Libvuln.
• Does the security database provide enough information to parse each entry into a
claircore.Vulnerability?
◦ Each entry must parse into a claircore.Vulnerability.
◦ Each Vulnerability must contain a package and a repository or distribution field.
• Will the Updater need to be configured at runtime?
◦ Your updater may implement the Configurable interface. Your matcher will have
its "Configuration" method called before use, giving you an opportunity for run
time configuration.
• What fields in a parsed claircore.Vulnerability will be present when indexing layer
artifacts.
◦ When implementing an updater you must keep in mind how
packages/distributions/repositories will be parsed during index. When indexing a
layer a common data model must exist between the possible
package/distribution/repository and the parsed Vulnerabilitie's
28 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
package/distribution/repository fields.
If you are having trouble figuring out these requirements do not hesitate to reach out to us
for help.
After you have taken the design points into consideration, you are ready to implement your
updater.
Typically you will create a new package named after the source you are adding support for.
Inside this package you can begin implementing the Updater and Updater Set Factory
interfaces.
Optionally you may implement the Configurable interface if you need runtime configuration.
It will undoubtly be helpful to look at the examples in the "ubuntu", "rhel", and "debian"
packages to get yourself started.
A package scanner is responsible for taking a claircore.Layer and parsing the contents for a
particular package database or set of files inside the provided tar archive. Once the target
files are located the package scanner should parse these files into claircore.Packages and
return a slice of these data structures.
Package scanning is context free, meaning no distribution classification has happened yet.
This is because manifests are made up of layers, and a layer which holds a package
database may not hold distribution information such as an os-release file. A package
scanner need only parse a target package database and return claircore.Packages.
Optionally, you may implement the Configurable Scanner if the scanner needs to perform
runtime configuration before use.
Keep in mind that its very common for distributions to utilize an existing package manager
such as RPM.
29 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
If this is the case there's a high likelihood that you can utilize the existing "rpm" or "dpkg"
package scanner implementations.
If your implemented Updater parses vulnerabilities with distribution information you will
likely need to implement a distribution scanner. Likewise, if your Updater parses
vulnerabilities with repository information (typical with language vulnerabilities) you will
likely need to implement a repository scanner.
The distribution scanner will parse the provided tar archive exhaustively searching for any
clue that this layer was derived from your distribution. If you identify that it is, you should
return a common distribution used by your Updater implementation. This ensures that
claircore can match the output of your distribution scanner with your parsed vulnerabilities.
Optionally, you may implement the Configurable Scanner if the scanner needs to perform
runtime configuration before use.
If your Updater parses vulnerabilities with a repository field you will likely want to implement
a repository scanner.
A repository scanner is used just like a distribution scanner however you will search for any
clues that a layer contains your repository and if so return a common data model identifying
the repository.
Optionally, you may implement the Configurable Scanner if the scanner needs to perform
runtime configuration before use.
30 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Implementing a Coalescer
As you may have noticed, the process of scanning a layer for packages, distribution, and
repository information is distinct and separate.
At some point, claircore will need to take all the context-free information returned from
layer scanners and create a complete view of the manifest. A coalescer performs this
computation.
It's unlikely you will need to implement your own coalescer. Claircore provides a default
"linux" coalescer which will work if your package database is rewritten when modified. For
example, if a Dockerfile's RUN command causes a change to to dpkg's /var/lib
/dpkg/status database, the resulting manifest will have a copy placed in the associated
layer.
However, if your package database does not fit into this model, implementing a coalescer
may be necessary.
If your distribution or language cannot utilize a default coalescer, you will need to implement
the Coalescer interface
31 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Libindex will take the Ecosystem and scan each layer with all provided scanners. When
Libindex is ready to coalesce the results of each scanner into an IndexReport the provided
coalescer is given the output of the configured scanners.
This allows Libindex to segment the input to the coalescing step to particular scanners that a
coalescer understands.
For instance, if we only wanted a (fictitious) Haskell coalescer to evaluate artifacts returned
from a (fictitious) Haskell package and repository scanner we would create an ecosystem
similar to:
// NewEcosystem provides the set of scanners and coalescers for the haskell
// ecosystem.
func NewEcosystem(ctx context.Context) *indexer.Ecosystem {
return &indexer.Ecosystem{
PackageScanners: func(ctx context.Context) ([]indexer.PackageScanner,
error) {
return []indexer.PackageScanner{haskellScanner()}, nil
},
DistributionScanners: func(ctx context.Context)
([]indexer.DistributionScanner, error) {
return []indexer.DistributionScanner{}, nil
},
RepositoryScanners: func(ctx context.Context)
([]indexer.RepositoryScanner, error) {
return []indexer.RepositoryScanner{}, nil
},
Coalescer: func(ctx context.Context) (indexer.Coalescer, error) {
return haskellCoalescer(), nil
},
}
}
This ensures that Libindex will only provide Haskell artifacts to the Haskell coalescer and
avoid calling the coalescer with rpm packages for example.
32 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
If your distribution uses an already implemented package manager such as "rpm" or "dpkg",
it's likely you will simply add your scanners to the existing ecosystem in one of those
packages.
Alternative Implementations
This how-to guide is a "perfect world" scenario.
Working on claircore has made us realize that this domain is a bit messy. Security trackers
are not developed with package managers in mind, security databases do not follow correct
specs, distribution maintainers spin their own tools, etc.
We understand that supporting your distribution or language may take some bending of
claircore's architecture and business logic. If this is the case, start a conversation with us. We
are open to design discussions.
Getting Help
At this point, you have implemented all the necessary components to integrate your
distribution or language with claircore.
If you struggle with the design phase or are getting stuck at the implementation phases do
not hesitate to reach out to us. Here are some links:
• Clair SIG
• Github Issues
• RedHat Issues
33 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Reference
The follow sections provide a reference for developing with Claircore. Important interfaces
and structs are outlined.
• Coalescer
• Configurable Scanner
• Distribution Scanner
• Ecosystem
• Libindex Store
• Matcher
• Package Scanner
• Remote Scanner
• Repository Scanner
• RPC Scanner
• Updater Set Factory
• Version Filter
• Versioned Scanner
• Resolver
34 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Coalescer
A coalescer must compute the final contents of a manifest given the artifacts found at each
layer.
35 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
ConfigurableScanner
A ConfigurableSanner is an optional interface a Scanner interface may implement. When
implemented, the scanner's Configure method will be called with a ConfigDeserializer
function. The Scanner may pass its config as an argument to the ConfigDeserializer
function to populate the struct.
36 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Distribution Scanner
A Distribution Scanner should identify any operating system distribution associated with the
provided layer. It is OK for no distribution information to be discovered.
37 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Ecosystem
An Ecosystem groups together scanners and coalescers which are often used together.
Ecosystems are usually defined in a go package that corresponds to a package manager,
such as dpkg . See dpkg/ecosystem.go for an example.
The Indexer will retrieve artifacts from the provided scanners and provide these scan
artifacts to the coalescer in the Ecosystem.
A typical ecosystem is "dpkg" which will use the "dpkg" package indexer,
the "os-release" distribution scanner and the "apt" repository scanner.
A Controller will scan layers with all scanners present in its configured
ecosystems.
38 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Index Report
An Index Report defines the contents of a manifest. An Index Report can be unpacked into a
slice of Index Records, each of which defines a package, distribution, repository tuple.
39 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
40 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Indexer Store
The indexer.Store interface defines all necessary persistence methods for Libindex to
provide its functionality.
type Setter interface { // PersistManifest must store the presence of a manifest and it's
layers into the system. // // Typically this will write into identity tables so later methods have
a foreign key // to reference and data integrity is applied. PersistManifest(ctx
context.Context, manifest claircore.Manifest) error // DeleteManifests removes the
manifests indicated by the passed digests // from the backing store.
DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)
41 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
} Setter interface provides the method set for required marking events, or registering
components, associated with an Index operation.
type Store interface { Setter Querier Indexer // Close frees any resources associated with the
Store. Close(context.Context) error } Store is an interface for dealing with objects libindex
needs to persist. Stores may be implemented per storage backend.
42 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
type Querier interface { // ManifestScanned returns whether the given manifest was
43 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
type Store interface { Setter Querier Indexer // Close frees any resources associated with the
Store. Close(context.Context) error } Store is an interface for dealing with objects libindex
needs to persist. Stores may be implemented per storage backend.
44 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
type Querier interface { // ManifestScanned returns whether the given manifest was
45 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
46 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Matcher Store
The datastore.MatcherStore interface defines all necessary persistence methods for
Libvuln to provide its functionality.
47 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
48 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
49 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
50 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
51 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
52 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
53 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
54 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
55 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
type Vulnerability interface { // get finds the vulnerabilities which match each package
provided in the packages array // this maybe a one to many relationship. each package is
assumed to have an ID. // a map of Package.ID => Vulnerabilities is returned. Get(ctx
context.Context, records []*claircore.IndexRecord, opts GetOpts) (map[string]
[]*claircore.Vulnerability, error) }
56 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Manifest
A Manifest is analogous to an OCI Image Manifest: it defines the order of layers and how to
retrieve the them.
57 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Matcher
A Matcher performs the heavy lifting of matching manifest contents to relevant
vulnerabilities. These implementations provide the smarts for understanding if a particular
artifact in a layer is vulnerable to a particular advisory in the database.
The Filter method is used to inform Libvuln the provided artifact is interesting. The
Query method tells Libvuln how to query the security advisory database. The Vulnerable
method reports whether the provided package is vulnerable to the provided vulnerability.
Typically, this would perform a version check between the artifact and the vulnerability in
question.
58 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Package Scanner
A Package Scanner should discover any packages found within the given layer. It is OK for to
discover no packages within a layer.
59 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
RemoteMatcher
RemoteMatcher is an additional interface a Matcher may implement to skip the database for
matching results and use an external API.
When called the interface can invoke the remote matcher using an HTTP API
to
fetch new vulnerabilities associated with the given IndexRecords.
The information retrieved from this interface won't be persisted into the
claircore database.
60 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Repository Scanner
A RepositoryScanner should identify any repositories discovered in the provided layer. It is
OK for the scanner to identify no repositories.
61 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Resolver
A Resolver is used to analyze and modify the post-coalesced index report. This is useful for
operations that need all context from an index report.
Resolvers are called at the end of the coalesce step when reports from
separate scanners are merged.
Any Resolvers' Resolve() methods are called (in no set order) at the end of the coalesce
step after reports from separate scanners are merged.
62 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
RPCScanner
RPCScanner is an optional interface a Scanner may implement. When implemented, the
scanner's Configure method will be called with a ConfigDeserializer function and an
HTTP client. The Scanner may pass its config as an argument to the ConfigDeserializer
function to populate the struct and use the HTTP client for any remote access necessary
during the scanning process.
63 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Updater
An Updater is responsible for performing run-time fetching and parsing of a security
database. The returned vulnerabilities will be written to claircore's database and used in
vulnerability matching.
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface
64 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
combining the method set of a Fetcher and a Parser and forces a Name() to be provided
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which
uniquely
identifies the new content.
type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the
contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should
assume contents are uncompressed and ready for parsing. Parse(ctx context.Context,
contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is
embedded into the Updater interface.
type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface
combining the method set of a Fetcher and a Parser and forces a Name() to be provided
65 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which
uniquely
identifies the new content.
type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the
contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should
assume contents are uncompressed and ready for parsing. Parse(ctx context.Context,
contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is
embedded into the Updater interface.
66 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
67 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed
If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.
68 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
69 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
UpdaterSetFactory
An UpdaterSetFactory is a factory for runtime construction and configuration for Updaters.
type UpdaterSet struct { // Has unexported fields. } UpdaterSet holds a deduplicated set of
updaters.
70 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
VersionFilter
VersionFilter is an additional interface a Matcher may implement. If implemented,
Libvuln will attempt to use the database and the normalized version field of a package to
filter vulnerabilities in the database. This is an opt-in optimization for when a package
manager's version scheme can be normalized into a claircore.Version .
71 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Versioned Scanner
A versioned scanner is typically embedded into other scanner types. It drives claircore's
ability to register and understand when updaters have been changed. Functions that want
to work with a generic scanner type should use a VersionedScanner .
Implementers of this interface must provide a unique name. Making changes to a scanner's
implementation must return a new value from Version . Implementers must return the
correct kind: one of "package", "distribution", or "repository"
72 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Vulnerability Report
A Vulnerability Report is a structure describing a specific manifest, its contents, and the
vulnerabilities affecting its contents.
Unpacking a report is done by mapping the keys in the PackageVulnerabilities field to the
data structures in other lookup maps.
For example:
73 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Contributors
These topics cover helpful tips for contributing to Claircore.
• Local Development
• Logging
• Misc
• Tests
74 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
How to contribute
The preferred workflow is to fork the quay/claircore repository, push a feature branch to
the new fork, then open a pull request. All pull requests should be targeted to the main
branch outside of exceptional circumstances.
Testing
As many tests as possible should run with the standard go test invocations. Adding the
special tag integration (e.g. go test -tags integration ./... ) will also run "integration"
tests. The project interprets "integration" tests to mean any test that would need external
resources, such as:
After at least one run with the integration tag, the tests should cache needed resources
and run as many tests as possible. See also the test/integration package.
Pull Requests
The Pull Request (PR) is the unit of code review. Claircore's review flow treats a feature
branch as a stack of patches to be applied. That is to say, the feature branch should be
rebased onto the target branch and have well-organized commits. Merge commits are
disallowed. If the author would prefer to not rewrite commit history while working through
reviews, fixup commits are the suggested way to achieve that. As many requirements as
possible are enforced by CI, like:
Please use the "draft" option if the branch is not ready. Please enable the "allow edits by
maintainers" option.
75 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
The maintainers may rebase, push, and merge contributors' branches. This may necessitate
doing a git reset <remote>/<branch> to update a local branch.
Conventions
Git commits should be formatted like "subject: summary" and avoid going over 80
characters per line. The "subject" is usually the package affected by the commit (like jar or
rhel -- the relative path isn't needed) but sometimes a broader category (like docs , all ,
or cicd ) is OK.
All the helper scripts should handle the "normal" convention ( origin is quay/claircore
and fork is one's personal fork) and the "British" convention ( origin is one's personal fork
and upstream is quay/claircore ).
76 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
The claircore project has switched from a git log based changelog to a git notes based
changelog.
This has the benefit of making the changelog more human-friendly, as it can have prose
describing changes now, but makes adding entries a bit more involved. A full understanding
of git notes is helpful for working with the changelog, but not required. If the reader has
worked with the notes feature before, the changelog entries are stored under the
changelog ref. For other users, there are some helper scripts in .github/scripts .
Git notes is a mechanism for storing additional information alongside commits without
modifying the commits. It does this by creating a ref full of files named after the commits,
with their contents being the notes. This scheme requires some special care and tooling --
see the documentation for more information.
Helper scripts
The primary helper script is changelog-edit . It allows a user to sync down notes, edit an
entry, or both. See the output of the h flag for more information.
The other script of interest is changelog-render , which can be used to render out the
changelog on demand, assuming the changelog notes have been pulled locally.
Formatting
Broadly, changelog entries should be formatted like commit messages without any trailers.
Entries are turned into list items, with the subject being the bullet point and the body of the
entry being the "body" of the item, or hidden behind details elements when using HTML-
enabled output.
The entries are almost always rendered as markdown, so using minimal markdown is OK.
Anything requiring excessive markdown is probably better served as documentation proper,
rather than a changelog entry.
77 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Local Development
A local development environment is implemented via docker-compose.
Usage
Several make targets are defined for working with the local development environment.
Tests
Several make targets are defined for working with tests.
78 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Logging
All the logging in claircore is done with zerolog via context.Context values. The zlog
package takes OpenTelemetry labels and attaches them to zerolog events.
This allows for claircore's logging to be used consistently throughout all the packages
without having unintended prints to stderr.
How to Log
Adding Context
ctx = zlog.ContextWithValues(ctx,
"component", "Example.Logger")
Logging style
Constant Messages
Zerolog emits lines when the Msg or Msgf methods are called. Project style is to not use
Msgf . Any variable data should be set as key-value pairs on the Event object.
Do this instead:
zlog.Info(ctx).
Time("time", time.Now()).
Msgf("done")
79 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Grammar
When noting the change during a chunk of work, make sure that the log messages scan as
visually similar. Usually, this means formatting messages into "${process} ${event}". For
example:
frob start
frob initialized
frob ready
frob success
frob done
starting to frob
initialized frobber
ready for frobbing
did frob
done with frobing
When handling an error, code should only log it if it does not propagate it. The code that
ultimately handles the error is responsible for deciding what to do with it. Logging and
returning ends up with the same message repeated multiple times in the logs.
Levels
Claircore attempts to have consistent leveled logging. The rules for figuring out what level to
use is:
• Panic
There's some occurrence that means the process won't work correctly.
• Fatal
• Error
Something unexpected occurred and the process can continue, but a human needs to
be notified. An error will be returned for this request.
80 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
• Warn
Something unexpected occurred and the process can continue. An error will be
returned for this request.
• Info
• Debug
Some information that may be useful to a developer. Examples include entering and
exiting functions, stepping through a function, or specific file paths used while work is
being done.
81 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
• URLs URLs in code should be annotated with a //doc:url directive comment. See the
the internal/cmd/mdbook-injecturls command for documentation on how the
preprocessor works. The list of keywords isn't an allowlist, so an invocation like the
following should list the ones actually used in the documentation using a command
like git grep injecturls -- :/docs/*.md .
82 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html
Tests
Tests in the claircore module may use various helpers underneath the test directory. Using
these packages outside of testing code is disallowed. Assert packages are disallowed; the
go-cmp package is the only external package helper allowed.
Tests that use external resources or generate test fixtures should be annotated according to
the integration package.
Caching
Tests using the integration package cache generated and downloaded assets into a
directory named clair-testing inside the directory reported by os.UserCacheDir . For
example, on a Linux system, the cache directory will be (in sh notation)
${XDG_CACHE_HOME-$HOME/.cache}/clair-testing .
83 of 83 11/30/2023, 5:26 PM