You are on page 1of 83

Claircore Documentation https://quay.github.io/claircore/print.

html

Claircore
Claircore is the engine behind the Clair v4 container security solution. The Claircore package
exports our domain models, interfaces necessary to plug into our business logic, and a
default set of implementations. This default set of implementations define our support
matrix and consists of the following distributions and languages:

• Ubuntu
• Debian
• RHEL
• Red Hat Container First content
• SUSE
• Oracle
• Alpine
• AWS Linux
• VMWare Photon
• Python
• Java
• Go
• Ruby

Claircore relies on PostgreSQL for its persistence and the library will handle migrations if
configured to do so.

The diagram below is a high level overview of Claircore's architecture.

Indexer

Image Manifest Libindex IndexReport Database

Matcher

Database IndexReport Libvuln VulnerabilityReport

When a claircore.Manifest is submitted to Libindex, the library will index its constituent parts
and create a report with its findings.

1 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

When a claircore.IndexReport is provided to Libvuln, the library will discover vulnerabilities


affecting it and generate a claircore.VulnerabilityReport.

2 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Getting Started
The following documentation helps a beginner learn to use Claircore.

• Libindex Usage
• Libvuln Usage

3 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Libindex Usage
Libindex is the Go package responsible for fetching container image layers, identifying
packages, distributions, and repositories within these layers, and computing a final
coalesced Index Report.

An Index Report is primarily used as input to Libvuln's vulnerability matching process.

Usage
Libindex is runtime constructed via the libindex.New method. New requires an
libindex.Opts struct.

Options

4 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package libindex // import "github.com/quay/claircore/libindex"

type Options struct {


// Store is the interface used to persist and retrieve results of indexing.
Store indexer.Store
// Locker provides system-wide locks. If the indexing work is distributed
the
// lock should be backed by a distributed store.
Locker LockSource
// FetchArena is an interface tied to the lifecycle of LibIndex to enable
management
// of the filesystem while separate processes are dealing with layers, for
example:
// you can reference count downloaded layer files to avoid racing.
FetchArena indexer.FetchArena
// ScanLockRetry specifies how often we should try to acquire a lock for
scanning a
// given manifest if lock is taken.
ScanLockRetry time.Duration
// LayerScanConcurrency specifies the number of layers to be scanned in
parallel.
LayerScanConcurrency int
// LayerFetchOpt is unused and kept here for backwards compatibility.
LayerFetchOpt interface{}
// NoLayerValidation controls whether layers are checked to actually be
// content-addressed. With this option toggled off, callers can trigger
// layers to be indexed repeatedly by changing the identifier in the
// manifest.
NoLayerValidation bool
// ControllerFactory provides an alternative method for creating a scanner
during libindex runtime
// if nil the default factory will be used. useful for testing purposes
ControllerFactory ControllerFactory
// Ecosystems a list of ecosystems to use which define which package
databases and coalescing methods we use
Ecosystems []*indexer.Ecosystem
// ScannerConfig holds functions that can be passed into configurable
// scanners. They're broken out by kind, and only used if a scanner
// implements the appropriate interface.
//
// Providing a function for a scanner that's not expecting it is not a
fatal
// error.
ScannerConfig struct {
Package, Dist, Repo, File map[string]func(interface{}) error
}
Resolvers []indexer.Resolver
}
Options are dependencies and options for constructing an instance of
libindex

The above outlines the relevant bits of the Options structure.

5 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Store is required needs to be an object that satisfies the indexer.Store interface.

Locker is required and needs to satisfy the LockSource interface.

FetchArena is required and needs to satify the FetchArena interface.

Providing a nil "Ecosystems" slice will supply the default set, instructing Libindex to index for
all supported content in a layer, and is typically desired.

Construction

Constructing Libindex is straight forward.

opts := new(libindex.Options)
// Populate with desired settings...
lib, err := libindex.New(ctx, opts, http.DefaultClient)
if err != nil {
panic(err)
}
defer lib.Close(ctx) // Remember to cleanup when done.

The constructing code should provide a valid Context tied to some lifetime.

Indexing

Indexing is the process of submitting a manifest to Libindex, fetching the manifest's layers,
indexing their contents, and coalescing a final Index Report.

Coalescing is the act of computing a final set of contents (packages, distributions, repos)
from a set of layers. Since layers maybe shared between many manifests, the final contents
of a manifest must be computed.

To perform an Index you must provide a claircore.Manifest data struture to the Index
method. The Manifest data structure describes an image manifest's layers and where they
can be fetched from.

m := new(claircore.Manifest)
// Populate somehow ...
ir, err := lib.Index(ctx, m)
if err != nil {
panic(err)
}

6 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

The Index method will block until an claircore.IndexReport is returned. The context should
be bound to some valid lifetime such as a request.

As the Indexer works on the manifest it will update its database throughout the process. You
may view the status of an index report via the "IndexReport" method.

ir, ok, err := lib.IndexReport(ctx, m.Hash)


if err != nil {
panic(err)
}

Libindex performs its work incrementally and saves state as it goes along. If Libindex
encounters an intermittent error during the index (for example, due to network failure while
fetching a layer), when the manifest is resubmitted only the layers not yet indexed will be
fetched and processed.

State

Libindex treats layers as content addressable. Once a layer identified by a particular hash is
indexed its contents are definitively known. A request to re-index a known layer results in
returning the previous successful response.

This comes in handy when dealing with base layers. The Ubuntu base layer is seen very
often across container registries. Treating this layer as content addressable precludes the
need to fetch and index the layer every time Libindex encounters it in a manifest.

There are times where re-indexing the same layer is necessary however. At the point where
Libindex realizes a new version of a component has not indexed a layer being submitted it
will perform the indexing operation.

A client must notice that Libindex has updated one of its components and subsequently
resubmit Manifests. The State endpoint is implemented for this reason.

Clients may query the State endpoint to receive an opaque string acting as a cookie,
identifying a unique state of Libindex. When a client sees this cookie change it should re-
submit manifests to Libindex to obtain a new index report.

7 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

state, err := lib.State(ctx)


if err != nil {
panic(err)
}
if state == prevState {
// Nothing to do.
return
}
// Otherwise, re-index manifest.

AffectedManifests

Libindex is capable of providing a client with all manifests affected by a set of vulnerabilities.
This functionality is designed for use with a notification mechanism.

var vulns []claircore.Vulnerability


affected, err := lib.AffectedManifests(ctx, vulns)
if err != nil {
panic(err)
}
for manifest, vulns := range affected.VulnerableManifests {
for _, vuln := range vulns {
fmt.Printf("vuln affecting manifest %s: %+v", manifest, vuln)
}
}

The slice of vulnerabilities returned for each manifest hash will be sorted by
claircore.NormalizedSeverity in "most severe" descending order.

8 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Libvuln Usage
Libvuln is the Go package responsible for keeping the database of vulnerabilities consistent,
matching container image contents with vulnerabilities, and reporting diffs between updates
of the same security database.

Usage
Libvuln is runtime constructed via the libvuln.New method. New requires a
libvuln.Opts struct.

Options

9 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package libvuln // import "github.com/quay/claircore/libvuln"

type Options struct {


// Store is the interface used to persist and retrieve vulnerabilites
// for of matching.
Store datastore.MatcherStore
// Locker provides system-wide locks for the updater subsystem. If the
// matching work is distributed the lock should be backed by a distributed
// store.
Locker LockSource
// An interval on which Libvuln will check for new security database
// updates.
//
// This duration will have jitter added to it, to help with smearing load
on
// installations.
UpdateInterval time.Duration
// A slice of strings representing which updaters libvuln will create.
//
// If nil all default UpdaterSets will be used.
//
// The following sets are supported:
// "alpine"
// "aws"
// "debian"
// "oracle"
// "photon"
// "pyupio"
// "rhel"
// "suse"
// "ubuntu"
UpdaterSets []string
// A list of out-of-tree updaters to run.
//
// This list will be merged with any defined UpdaterSets.
//
// If you desire no updaters to run do not add an updater
// into this slice.
Updaters []driver.Updater
// A slice of strings representing which
// matchers will be used.
//
// If nil all default Matchers will be used
//
// The following names are supported by default:
// "alpine"
// "aws"
// "debian"
// "oracle"
// "photon"
// "python"
// "rhel"

10 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

// "suse"
// "ubuntu"
MatcherNames []string

// Config holds configuration blocks for MatcherFactories and Matchers,


// keyed by name.
MatcherConfigs map[string]driver.MatcherConfigUnmarshaler

// A list of out-of-tree matchers you'd like libvuln to


// use.
//
// This list will me merged with the default matchers.
Matchers []driver.Matcher

// Enrichers is a slice of enrichers to use with all VulnerabilityReport


// requests.
Enrichers []driver.Enricher

// UpdateWorkers controls the number of update workers running


concurrently.
// If less than or equal to zero, a sensible default will be used.
UpdateWorkers int

// UpdateRetention controls the number of updates to retain between


// garbage collection periods.
//
// The lowest possible value is 2 in order to compare updates for
notification
// purposes.
UpdateRetention int

// If set to true, there will not be a goroutine launched to periodically


// run updaters.
DisableBackgroundUpdates bool

// UpdaterConfigs is a map of functions for configuration of Updaters.


UpdaterConfigs map[string]driver.ConfigUnmarshaler

// Client is an http.Client for use by all updaters.


//
// Must be set.
Client *http.Client
}

The above outlines the relevant bits of the Opts structure.

Construction

Constructing Libvuln is straight forward.

11 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

opts := new(libvuln.Options)
// Populate with desired settings...
lib, err := libvuln.New(ctx, opts)
if err != nil {
panic(err)
}
defer lib.Close(ctx)

The constructing code should provide a valid Context tied to some lifetime.

On construction, New will block until the security databases are initialized. Expect some
delay before this method returns.

Scanning

Scanning is the process of taking a claircore.IndexReport comprised of a Manifest's


content and determining which vulnerabilities affect the Manifest. A
claircore.VulnerabilityReport will be returned with these details.

m := new(claircore.Manifest)
// Populate somehow ...
ir, err := indexer.Index(ctx, m)
if err != nil {
panic(err)
}
vr, err := lib.Scan(ctx, ir)
if err != nil {
panic(err)
}

In the above example, Libindex is used to generate a claircore.IndexReport . The index


report is then provided to Libvuln and a subsequent vulnerability report identifying any
vulnerabilities affecting the manifest is returned.

Updates API

By default, Libvuln manages a set of long running updaters responsible for periodically
fetching and loading new advisory contents into its database. The Updates API allows a client
to view and manipulate aspects of the update operations that updaters perform.

In this getting started guide, we will only cover the two methods most interesting to new
users.

12 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

UpdateOperations

This API provides a list of recent update operations performed by implemented updaters.
The UpdateOperation slice returned will be sorted by latest timestamp descending.

ops, err := lib.UpdateOperations(ctx, `updater`)


if err != nil {
panic(err)
}
for updater, ops := range ops {
fmt.Printf("ops for updater %s, %+v", updater, ops)
}

UpdateDiff

Mostly used by the Clair v4 notification subsystem, this endpoint will provide the caller with
any removed or added vulnerabilities between two update operations. Typically a diff takes
places against two versions of the same data source. This is useful to inform downstream
applications what new vulnerabilities have entered the system.

ops, err := lib.UpdateOperations(ctx, `updater`)


if err != nil {
panic(err)
}
for upd, ops := range ops {
fmt.Printf("updater: %v", upd)
diff, err := lib.UpdateDiff(ctx, ops[1].Ref, ops[0].Ref)
if err != nil {
panic(err)
}
for _, vuln := range diff.Added {
fmt.Printf("vuln %+v added in %v", vuln, diff.Cur.Ref)
}
}

13 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Concepts
The following sections give a conceptual overview of how Claircore works internally.

• Vulnerability Matching
• Indexer Architecture
• Matching Architecture
• Severity Mapping

14 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Vulnerability Matching
The following describes a successful scan.

1. Updaters have ran either in the background on an interval or an offline loader has
been ran.
2. A Manifest is provided to Libindex. Libindex fetches all the layers, runs all scanner
types on each layer, persists all artifacts found in each layer, and computes an
IndexReport.
3. A IndexReport is provided to Libvuln.
4. Libvuln creates a stream of IndexRecord structs from the IndexReport and
concurrently streams these structs to each configured Matcher.
5. Libvuln computes a VulnerabilityReport aggregating all vulnerabilities discovered by all
configured Matcher implementations.
6. Sometime later the security advisory database is updated and a new request to Libvuln
will present updated vulnerability data.

15 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Indexer
claircore/indexer

The Indexer package performs Libindex's heavy lifting. It is responsible for retreiving
Manifest layers, parsing the contents of each layer, and computing an IndexReport.

To perform this action in incremental steps the Indexer is implemented as a finite state
machine. At each state transition the Indexer persists an updated IndexReport to its
datastore.

States
The following diagram expresses the possible states of the Indexer:

16 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

CheckManifest

Unindexed

FetchLayers

ScanLayers

Indexed Coalesce

IndexManifest

IndexFinished

Data Model
The Indexer data model focuses on content addressable hashes as primary keys, the
deduplication of package/distribution/repostitory information, and the recording of scan
artifacts. Scan artifacts are unique artifacts found within a layer which point to a
deduplicated general package/distribution/repository record.

The following diagram outlines the current Indexer data model.

17 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

ManifestLayer

cached result IndexReport

Manifest

ScannedManifest

Layer

ScannedLayer

Scanner

ManifestIndex

18 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

TYPE
TYPE_ScanArtifact

Note that TYPE stands in for each of the Indexer types (i.e. Package , Repository , etc.).

HTTP Resources
Indexers as currently built may make network requests. This is an outstanding issue. The
following are the URLs used.

• https://search.maven.org/solrsearch/select
• https://catalog.redhat.com/api/containers/
• https://access.redhat.com/security/data/metrics/repository-to-cpe.json
• https://access.redhat.com/security/data/metrics/container-name-repos-map.json

19 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Matcher Architecture
claircore/internal/matcher claircore/libvuln/driver

The Matcher architecture is based on a data flow application.

Libvuln.Scan

Matcher

Filter interested packages


RemoteMatcher

Filter interested packages

Retrive vulnerabilty information

Make API calls

Check versions in-database

Decide vulnerability

Deicide vulnerability

Merge into VulnerabilityReport

When Libvuln's Scan method is called with an IndexReport it will begin the process of
matching container contents with vulnerabilities.

Each configured Matcher will be instantiated concurrently. Depending on the interfaces the
Matcher implements, one of the possible data flows will occur in the diagram above.

The provided IndexReport will be unpacked into a stream of IndexRecord structs. Each
Matcher will evaluate each record in the stream and determine if the IndexRecord is
vulnerable to a security advisory in their responsible databases.

20 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Once each Matcher returns the set of vulnerabities, Libvuln will merge the results into a
VulnerabilityReport and return this to the client.

HTTP Resources
NOTE: Remote matchers are being considered for removal.

"Remote matchers" may make HTTP requests during the matcher flow. These requests are
time-bound and errors are not reported. The following are the URLs used.

21 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Severity Mapping
Claircore will normalize a security databases's severity string to a set of defined values.
Clients may use the NormalizedSeverity field on a claircore.Vulnerability to react to
vulnerability severities without needing to know each security database's severity strings. All
strings used in the mapping tables are identical to the strings found within the relevant
security database.

Claircore Severity Strings


The following are severity strings Claircore will normalize others to. Clients can guarantee
one of these strings will be associated with a claircore.Vulnerability.

• Unknown
• Negligible
• Low
• Medium
• High
• Critical

Alpine Mapping
The Alpine SecDB database does not provide severity information. All vulnerability severities
will be Unknown.

Alpine Severity Claircore Severity


* Unknown

AWS Mapping
The AWS UpdateInfo database provides severity information.

AWS Severity Claircore Severity


low Low

22 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

AWS Severity Claircore Severity


medium Medium
important High
critical Critical

Debian Mapping
The Debian security tracker data provides severity information.

Debian Severity Claircore Severity


unimportant Low
low Medium
medium High
high Critical
* Unknown

Oracle Mapping
The Oracle OVAL database provides severity information.

Oracle Severity Claircore Severity


N/A Unknown
LOW Low
MODERATE Medium
IMPORTANT High
CRITICAL Critical

RHEL Mapping
The RHEL OVAL database provides severity information.

RHEL Severity Claircore Severity

23 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

RHEL Severity Claircore Severity


None Unknown
Low Low
Moderate Medium
Important High
Critical Critical

SUSE Mapping
The SUSE OVAL database provides severity information.

SUSE Severity Claircore Severity


None Unknown
Low Low
Moderate Medium
Important High
Critical Critical

Ubuntu Mapping
The Ubuntu OVAL database provides severity information.

Ubuntu Severity Claircore Severity


Untriaged Unknown
Negligible Negligible
Low Low
Medium Medium
High High
Critical Critical

Photon Mapping

24 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

The Photon OVAL database provides severity information.

Photon Severity Claircore Severity


Low Low
Moderate Medium
Important High
Critical Critical

OSV Mapping
OSV provides severity information via CVSS vectors, when applicable. These are normalized
according to the NVD qualitative rating scale. If both v3 and v2 vectors are present, v3 is
preferred.

CVSSv3

Base Score Claircore Severity


0.0 Negligible
0.1-3.9 Low
4.0-6.9 Medium
7.0-8.9 High
9.0-10.0 Critical

CVSSv2

Base Score Claircore Severity


0.0-3.9 Low
4.0-6.9 Medium
7.0-10.0 High

25 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Updaters and Defaults


The default updaters are tracked in updater/defaults/defaults.go .

HTTP Resources
The following are the HTTP hosts and paths that Clair will attempt to talk to in a default
configuration. This list is non-exhaustive, as some servers will issue redirects and some
request URLs are constructed dynamically.

• https://secdb.alpinelinux.org/
• http://repo.us-west-2.amazonaws.com/2018.03/updates/x86_64/mirror.list
• https://cdn.amazonlinux.com/2/core/latest/x86_64/mirror.list
• https://cdn.amazonlinux.com/al2023/core/mirrors/latest/x86_64/mirror.list
• https://deb.debian.org/
• https://security-tracker.debian.org/tracker/data/json
• https://nvd.nist.gov/feeds/json/cve/1.1/
• https://linux.oracle.com/security/oval/com.oracle.elsa-*.xml.bz2
• https://packages.vmware.com/photon/photon_oval_definitions/
• https://access.redhat.com/security/data/metrics/cvemap.xml
• https://access.redhat.com/security/cve/
• https://access.redhat.com/security/data/oval/v2/PULP_MANIFEST
• https://support.novell.com/security/oval/
• https://api.launchpad.net/1.0/
• https://security-metadata.canonical.com/oval/com.ubuntu.*.cve.oval.xml
• https://osv-vulnerabilities.storage.googleapis.com/

26 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

How Tos
The following sections provide instructions on accomplish specific goals in Claircore.

• Adding Distribution Or Language Support

27 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Adding Distribution And Language Support


Note: If terms in this document sound foreign check out Getting Started to acquaint yourself with
"indexing", "scanning", and "matching"

The claircore team is always open to adding more distributions and languages to the library.

Generally, distributions or languages must provide a security tracker.

All officially supported distributions and languages provide a database of security


vulnerabilities.

These databases are maintained by the distribution or language developers and reflect up-
to-date CVE and advisory data for their packages.

If your distribution or language does not provide a security tracker or piggy-backs off


another distribution's start an issue in our Github issue tracker to discuss further.

Implementing an Updater
The first step to adding your distribution or language to claircore is getting your security
tracker's data ingested by Libvuln.

The Updater interfaces are responsible for this task.

An implementer must consider several design points:

• Does the security database provide enough information to parse each entry into a
claircore.Vulnerability?
◦ Each entry must parse into a claircore.Vulnerability.
◦ Each Vulnerability must contain a package and a repository or distribution field.
• Will the Updater need to be configured at runtime?
◦ Your updater may implement the Configurable interface. Your matcher will have
its "Configuration" method called before use, giving you an opportunity for run
time configuration.
• What fields in a parsed claircore.Vulnerability will be present when indexing layer
artifacts.
◦ When implementing an updater you must keep in mind how
packages/distributions/repositories will be parsed during index. When indexing a
layer a common data model must exist between the possible
package/distribution/repository and the parsed Vulnerabilitie's

28 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package/distribution/repository fields.

If you are having trouble figuring out these requirements do not hesitate to reach out to us
for help.

After you have taken the design points into consideration, you are ready to implement your
updater.

Typically you will create a new package named after the source you are adding support for.

Inside this package you can begin implementing the Updater and Updater Set Factory
interfaces.

Optionally you may implement the Configurable interface if you need runtime configuration.

It will undoubtly be helpful to look at the examples in the "ubuntu", "rhel", and "debian"
packages to get yourself started.

Implementing a Package Scanner


At this point you hopefully have your Updater working, writing vulnerability data into
Libvuln's database.

We can now move our attention to package scanning.

A package scanner is responsible for taking a claircore.Layer and parsing the contents for a
particular package database or set of files inside the provided tar archive. Once the target
files are located the package scanner should parse these files into claircore.Packages and
return a slice of these data structures.

Package scanning is context free, meaning no distribution classification has happened yet.
This is because manifests are made up of layers, and a layer which holds a package
database may not hold distribution information such as an os-release file. A package
scanner need only parse a target package database and return claircore.Packages.

You need to implement the Package Scanner interface to achieve this.

Optionally, you may implement the Configurable Scanner if the scanner needs to perform
runtime configuration before use.

Keep in mind that its very common for distributions to utilize an existing package manager
such as RPM.

29 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

If this is the case there's a high likelihood that you can utilize the existing "rpm" or "dpkg"
package scanner implementations.

Implementing a Distribution Scanner


Once the package scanner is implemented, tested, and working you can begin implementing
a Distribution Scanner.

Implementing a distribution scanner is a design choice. Distributions and repositories are


the way claircore matches packages to vulnerabilities.

If your implemented Updater parses vulnerabilities with distribution information you will
likely need to implement a distribution scanner. Likewise, if your Updater parses
vulnerabilities with repository information (typical with language vulnerabilities) you will
likely need to implement a repository scanner.

A distribution scanner, like a package scanner, is provided a claircore.Layer.

The distribution scanner will parse the provided tar archive exhaustively searching for any
clue that this layer was derived from your distribution. If you identify that it is, you should
return a common distribution used by your Updater implementation. This ensures that
claircore can match the output of your distribution scanner with your parsed vulnerabilities.

Optionally, you may implement the Configurable Scanner if the scanner needs to perform
runtime configuration before use.

Implementing a Repository Scanner


As mentioned above, implementing a repository scanner is a design choice, often times
applicable for language package managers.

If your Updater parses vulnerabilities with a repository field you will likely want to implement
a repository scanner.

A repository scanner is used just like a distribution scanner however you will search for any
clues that a layer contains your repository and if so return a common data model identifying
the repository.

Optionally, you may implement the Configurable Scanner if the scanner needs to perform
runtime configuration before use.

30 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Implementing a Coalescer
As you may have noticed, the process of scanning a layer for packages, distribution, and
repository information is distinct and separate.

At some point, claircore will need to take all the context-free information returned from
layer scanners and create a complete view of the manifest. A coalescer performs this
computation.

It's unlikely you will need to implement your own coalescer. Claircore provides a default
"linux" coalescer which will work if your package database is rewritten when modified. For
example, if a Dockerfile's RUN command causes a change to to dpkg's /var/lib
/dpkg/status database, the resulting manifest will have a copy placed in the associated
layer.

However, if your package database does not fit into this model, implementing a coalescer
may be necessary.

To implement a coalescer, several details must be understood:

• Each layer only provides a "piece" of the final manifest.


◦ Because manifests are comprised of multiple copy-on-write layers, some layers
may contain package information, distribution information, repository
information, any combination of those, or no information at all.
• An OS may have a "dist-upgrade" performed and the implications of this on the
package management system is distribution or language dependent.
◦ The coalescer must deal with distribution upgrades in a sane way. If your
distribution or language does a dist-upgrade, are all packages bumped? Are they
simply left alone? The coalescer must understand what happens and compute the
final manifest's content correctly.
• Packages may be removed and added between layers.
◦ When the package database is a regular file on disk, this case is simpler: the
database file found in the most recent layer holds the ultimate set of packages for
all previous layers. However, in the case where the package database is realized
by adding and removing files on disk it becomes trickier. Claircore has no special
handling of whiteout files, currently. We will address this in upcoming releases.

If your distribution or language cannot utilize a default coalescer, you will need to implement
the Coalescer interface

31 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Implementing or Adding To An Ecosystem


An Ecosystem provides a set of coalescers, package scanners, distribution scanners, and
repository scanners to Libindex at the time of indexing.

Libindex will take the Ecosystem and scan each layer with all provided scanners. When
Libindex is ready to coalesce the results of each scanner into an IndexReport the provided
coalescer is given the output of the configured scanners.

This allows Libindex to segment the input to the coalescing step to particular scanners that a
coalescer understands.

For instance, if we only wanted a (fictitious) Haskell coalescer to evaluate artifacts returned
from a (fictitious) Haskell package and repository scanner we would create an ecosystem
similar to:

// HaskellScanner returns a configured PackageScanner.


func haskellScanner() indexer.PackageScanner { return nil }

// HaskellCoalescer returns a configured Coalescer.


func haskellCoalescer() indexer.Coalescer { return nil }

// NewEcosystem provides the set of scanners and coalescers for the haskell
// ecosystem.
func NewEcosystem(ctx context.Context) *indexer.Ecosystem {
return &indexer.Ecosystem{
PackageScanners: func(ctx context.Context) ([]indexer.PackageScanner,
error) {
return []indexer.PackageScanner{haskellScanner()}, nil
},
DistributionScanners: func(ctx context.Context)
([]indexer.DistributionScanner, error) {
return []indexer.DistributionScanner{}, nil
},
RepositoryScanners: func(ctx context.Context)
([]indexer.RepositoryScanner, error) {
return []indexer.RepositoryScanner{}, nil
},
Coalescer: func(ctx context.Context) (indexer.Coalescer, error) {
return haskellCoalescer(), nil
},
}
}

This ensures that Libindex will only provide Haskell artifacts to the Haskell coalescer and
avoid calling the coalescer with rpm packages for example.

32 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

If your distribution uses an already implemented package manager such as "rpm" or "dpkg",
it's likely you will simply add your scanners to the existing ecosystem in one of those
packages.

Alternative Implementations
This how-to guide is a "perfect world" scenario.

Working on claircore has made us realize that this domain is a bit messy. Security trackers
are not developed with package managers in mind, security databases do not follow correct
specs, distribution maintainers spin their own tools, etc.

We understand that supporting your distribution or language may take some bending of
claircore's architecture and business logic. If this is the case, start a conversation with us. We
are open to design discussions.

Getting Help
At this point, you have implemented all the necessary components to integrate your
distribution or language with claircore.

If you struggle with the design phase or are getting stuck at the implementation phases do
not hesitate to reach out to us. Here are some links:

• Clair SIG
• Github Issues
• RedHat Issues

33 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Reference
The follow sections provide a reference for developing with Claircore. Important interfaces
and structs are outlined.

• Coalescer
• Configurable Scanner
• Distribution Scanner
• Ecosystem
• Libindex Store
• Matcher
• Package Scanner
• Remote Scanner
• Repository Scanner
• RPC Scanner
• Updater Set Factory
• Version Filter
• Versioned Scanner
• Resolver

34 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Coalescer
A coalescer must compute the final contents of a manifest given the artifacts found at each
layer.

package indexer // import "github.com/quay/claircore/indexer"

type Coalescer interface {


Coalesce(ctx context.Context, artifacts []*LayerArtifacts)
(*claircore.IndexReport, error)
}
Coalescer takes a set of layers and creates coalesced IndexReport.

A coalesced IndexReport should provide only the packages present in the


final container image once all layers were applied.

package indexer // import "github.com/quay/claircore/indexer"

type Coalescer interface {


Coalesce(ctx context.Context, artifacts []*LayerArtifacts)
(*claircore.IndexReport, error)
}
Coalescer takes a set of layers and creates coalesced IndexReport.

A coalesced IndexReport should provide only the packages present in the


final container image once all layers were applied.

package indexer // import "github.com/quay/claircore/indexer"

type LayerArtifacts struct { Hash claircore.Digest Pkgs []*claircore.Package Dist


[]*claircore.Distribution // each layer can only have a single distribution Repos
[]*claircore.Repository Files []claircore.File } LayerArifact aggregates the artifacts found
within a layer.

A `Coalsecer` implementation is free to determine this computation given the


artifacts found in a layer. A `Coalescer` is called with a slice of
`LayerArtifacts` structs. The manifest's layer ordering is preserved in the
provided slice.

35 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

ConfigurableScanner
A ConfigurableSanner is an optional interface a Scanner interface may implement. When
implemented, the scanner's Configure method will be called with a ConfigDeserializer
function. The Scanner may pass its config as an argument to the ConfigDeserializer
function to populate the struct.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigDeserializer func(interface{}) error


ConfigDeserializer can be thought of as an Unmarshal function with the byte
slice provided.

This will typically be something like (*json.Decoder).Decode.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigDeserializer func(interface{}) error


ConfigDeserializer can be thought of as an Unmarshal function with the byte
slice provided.

This will typically be something like (*json.Decoder).Decode.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigurableScanner interface { Configure(context.Context, ConfigDeserializer) error }


ConfigurableScanner is an interface scanners can implement to receive configuration.

36 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Distribution Scanner
A Distribution Scanner should identify any operating system distribution associated with the
provided layer. It is OK for no distribution information to be discovered.

package indexer // import "github.com/quay/claircore/indexer"

type DistributionScanner interface {


VersionedScanner
Scan(context.Context, *claircore.Layer) ([]*claircore.Distribution, error)
}
DistributionScanner reports the Distributions found in a given layer.

37 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Ecosystem
An Ecosystem groups together scanners and coalescers which are often used together.
Ecosystems are usually defined in a go package that corresponds to a package manager,
such as dpkg . See dpkg/ecosystem.go for an example.

The Indexer will retrieve artifacts from the provided scanners and provide these scan
artifacts to the coalescer in the Ecosystem.

package indexer // import "github.com/quay/claircore/indexer"

type Ecosystem struct {


PackageScanners func(ctx context.Context) ([]PackageScanner, error)
DistributionScanners func(ctx context.Context) ([]DistributionScanner,
error)
RepositoryScanners func(ctx context.Context) ([]RepositoryScanner, error)
FileScanners func(ctx context.Context) ([]FileScanner, error)
Coalescer func(ctx context.Context) (Coalescer, error)
Name string
}
Ecosystems group together scanners and a Coalescer which are commonly used
together.

A typical ecosystem is "dpkg" which will use the "dpkg" package indexer,
the "os-release" distribution scanner and the "apt" repository scanner.

A Controller will scan layers with all scanners present in its configured
ecosystems.

38 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Index Report
An Index Report defines the contents of a manifest. An Index Report can be unpacked into a
slice of Index Records, each of which defines a package, distribution, repository tuple.

package claircore // import "github.com/quay/claircore"

type IndexReport struct {


// the manifest hash this IndexReport is describing
Hash Digest `json:"manifest_hash"`
// the current state of the index operation
State string `json:"state"`
// all discovered packages in this manifest key'd by package id
Packages map[string]*Package `json:"packages"`
// all discovered distributions in this manifest key'd by distribution id
Distributions map[string]*Distribution `json:"distributions"`
// all discovered repositories in this manifest key'd by repository id
Repositories map[string]*Repository `json:"repository"`
// a list of environment details a package was discovered in key'd by
package id
Environments map[string][]*Environment `json:"environments"`
// whether the index operation finished successfully
Success bool `json:"success"`
// an error string in the case the index did not succeed
Err string `json:"err"`
// Files doesn't end up in the json report but needs to be available at
post-coalesce
Files map[string]File `json:"-"`
}
IndexReport provides a database for discovered artifacts in an image.

IndexReports make heavy usage of lookup maps to associate information


without repetition.

func (report *IndexReport) IndexRecords() []*IndexRecord

39 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package claircore // import "github.com/quay/claircore"

type IndexReport struct {


// the manifest hash this IndexReport is describing
Hash Digest `json:"manifest_hash"`
// the current state of the index operation
State string `json:"state"`
// all discovered packages in this manifest key'd by package id
Packages map[string]*Package `json:"packages"`
// all discovered distributions in this manifest key'd by distribution id
Distributions map[string]*Distribution `json:"distributions"`
// all discovered repositories in this manifest key'd by repository id
Repositories map[string]*Repository `json:"repository"`
// a list of environment details a package was discovered in key'd by
package id
Environments map[string][]*Environment `json:"environments"`
// whether the index operation finished successfully
Success bool `json:"success"`
// an error string in the case the index did not succeed
Err string `json:"err"`
// Files doesn't end up in the json report but needs to be available at
post-coalesce
Files map[string]File `json:"-"`
}
IndexReport provides a database for discovered artifacts in an image.

IndexReports make heavy usage of lookup maps to associate information


without repetition.

func (report *IndexReport) IndexRecords() []*IndexRecord

package claircore // import "github.com/quay/claircore"

type IndexRecord struct { Package *Package Distribution *Distribution Repository


*Repository } IndexRecord is an entry in the IndexReport.

IndexRecords provide full access to contextual package structures such as


Distribution and Repository.

A list of these can be thought of as an "unpacked" IndexReport

40 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Indexer Store
The indexer.Store interface defines all necessary persistence methods for Libindex to
provide its functionality.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface {


Setter
Querier
Indexer
// Close frees any resources associated with the Store.
Close(context.Context) error
}
Store is an interface for dealing with objects libindex needs to persist.
Stores may be implemented per storage backend.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface {


Setter
Querier
Indexer
// Close frees any resources associated with the Store.
Close(context.Context) error
}
Store is an interface for dealing with objects libindex needs to persist.
Stores may be implemented per storage backend.

package indexer // import "github.com/quay/claircore/indexer"

type Setter interface { // PersistManifest must store the presence of a manifest and it's
layers into the system. // // Typically this will write into identity tables so later methods have
a foreign key // to reference and data integrity is applied. PersistManifest(ctx
context.Context, manifest claircore.Manifest) error // DeleteManifests removes the
manifests indicated by the passed digests // from the backing store.
DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest, error)

41 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

// SetLayerScanned marks the provided layer hash successfully scanned by the


provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the same
arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr
VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the persistence
layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for clients.
this method should be
// used to communicate scanning state updates. to signal the scan has
completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the
manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in the
provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs
VersionedScanners) error

} Setter interface provides the method set for required marking events, or registering
components, associated with an Index operation.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface { Setter Querier Indexer // Close frees any resources associated with the
Store. Close(context.Context) error } Store is an interface for dealing with objects libindex
needs to persist. Stores may be implemented per storage backend.

42 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package indexer // import "github.com/quay/claircore/indexer"

type Setter interface {


// PersistManifest must store the presence of a manifest and it's layers
into the system.
//
// Typically this will write into identity tables so later methods have a
foreign key
// to reference and data integrity is applied.
PersistManifest(ctx context.Context, manifest claircore.Manifest) error
// DeleteManifests removes the manifests indicated by the passed digests
// from the backing store.
DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest,
error)

// SetLayerScanned marks the provided layer hash successfully scanned by


the provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the
same arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr
VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the
persistence layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for
clients. this method should be
// used to communicate scanning state updates. to signal the scan has
completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the
manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in
the provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs
VersionedScanners) error
}
Setter interface provides the method set for required marking events,
or registering components, associated with an Index operation.

package indexer // import "github.com/quay/claircore/indexer"

type Querier interface { // ManifestScanned returns whether the given manifest was

43 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

scanned by the provided scanners. ManifestScanned(ctx context.Context, hash


claircore.Digest, scnrs VersionedScanners) (bool, error) // LayerScanned returns whether the
given layer was scanned by the provided scanner. LayerScanned(ctx context.Context, hash
claircore.Digest, scnr VersionedScanner) (bool, error) // PackagesByLayer gets all the
packages found in a layer limited by the provided scanners. PackagesByLayer(ctx
context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Package, error)
// DistributionsByLayer gets all the distributions found in a layer limited by the provided
scanners. DistributionsByLayer(ctx context.Context, hash claircore.Digest, scnrs
VersionedScanners) ([]*claircore.Distribution, error) // RepositoriesByLayer gets all the
repositories found in a layer limited by the provided scanners. RepositoriesByLayer(ctx
context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Repository,
error) // FilesByLayer gets all the interesting files found in a layer limited by the provided
scanners. FilesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners)
([]claircore.File, error) // IndexReport attempts to retrieve a persisted IndexReport.
IndexReport(ctx context.Context, hash claircore.Digest) (*claircore.IndexReport, bool, error)
// AffectedManifests returns a list of manifest digests which the target vulnerability // affects.
AffectedManifests(ctx context.Context, v claircore.Vulnerability, f
claircore.CheckVulnernableFunc) ([]claircore.Digest, error) } Querier interface provides the
method set to ascertain indexed artifacts and query whether a layer or manifest has been
scanned.

package indexer // import "github.com/quay/claircore/indexer"

type Store interface { Setter Querier Indexer // Close frees any resources associated with the
Store. Close(context.Context) error } Store is an interface for dealing with objects libindex
needs to persist. Stores may be implemented per storage backend.

44 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package indexer // import "github.com/quay/claircore/indexer"

type Setter interface {


// PersistManifest must store the presence of a manifest and it's layers
into the system.
//
// Typically this will write into identity tables so later methods have a
foreign key
// to reference and data integrity is applied.
PersistManifest(ctx context.Context, manifest claircore.Manifest) error
// DeleteManifests removes the manifests indicated by the passed digests
// from the backing store.
DeleteManifests(context.Context, ...claircore.Digest) ([]claircore.Digest,
error)

// SetLayerScanned marks the provided layer hash successfully scanned by


the provided versioned scanner.
//
// After this method is returned a call to Querier.LayerScanned with the
same arguments must return true.
SetLayerScanned(ctx context.Context, hash claircore.Digest, scnr
VersionedScanner) error
// RegisterPackageScanners registers the provided scanners with the
persistence layer.
RegisterScanners(ctx context.Context, scnrs VersionedScanners) error
// SetIndexReport persists the current state of the IndexReport.
//
// IndexReports maybe in intermediate states to provide feedback for
clients. this method should be
// used to communicate scanning state updates. to signal the scan has
completely successfully
// see SetIndexFinished.
SetIndexReport(context.Context, *claircore.IndexReport) error
// SetIndexFinished marks a scan successfully completed.
//
// After this method returns a call to Querier.ManifestScanned with the
manifest hash represted in the provided IndexReport
// must return true.
//
// Also a call to Querier.IndexReport with the manifest hash represted in
the provided IndexReport must return the IndexReport
// in finished state.
SetIndexFinished(ctx context.Context, sr *claircore.IndexReport, scnrs
VersionedScanners) error
}
Setter interface provides the method set for required marking events,
or registering components, associated with an Index operation.

package indexer // import "github.com/quay/claircore/indexer"

type Querier interface { // ManifestScanned returns whether the given manifest was

45 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

scanned by the provided scanners. ManifestScanned(ctx context.Context, hash


claircore.Digest, scnrs VersionedScanners) (bool, error) // LayerScanned returns whether the
given layer was scanned by the provided scanner. LayerScanned(ctx context.Context, hash
claircore.Digest, scnr VersionedScanner) (bool, error) // PackagesByLayer gets all the
packages found in a layer limited by the provided scanners. PackagesByLayer(ctx
context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Package, error)
// DistributionsByLayer gets all the distributions found in a layer limited by the provided
scanners. DistributionsByLayer(ctx context.Context, hash claircore.Digest, scnrs
VersionedScanners) ([]*claircore.Distribution, error) // RepositoriesByLayer gets all the
repositories found in a layer limited by the provided scanners. RepositoriesByLayer(ctx
context.Context, hash claircore.Digest, scnrs VersionedScanners) ([]*claircore.Repository,
error) // FilesByLayer gets all the interesting files found in a layer limited by the provided
scanners. FilesByLayer(ctx context.Context, hash claircore.Digest, scnrs VersionedScanners)
([]claircore.File, error) // IndexReport attempts to retrieve a persisted IndexReport.
IndexReport(ctx context.Context, hash claircore.Digest) (*claircore.IndexReport, bool, error)
// AffectedManifests returns a list of manifest digests which the target vulnerability // affects.
AffectedManifests(ctx context.Context, v claircore.Vulnerability, f
claircore.CheckVulnernableFunc) ([]claircore.Digest, error) } Querier interface provides the
method set to ascertain indexed artifacts and query whether a layer or manifest has been
scanned.

package indexer // import "github.com/quay/claircore/indexer"

type Indexer interface {


// IndexPackages indexes a package into the persistence layer.
IndexPackages(ctx context.Context, pkgs []*claircore.Package, layer
*claircore.Layer, scnr VersionedScanner) error
// IndexDistributions indexes distributions into the persistence layer.
IndexDistributions(ctx context.Context, dists []*claircore.Distribution,
layer *claircore.Layer, scnr VersionedScanner) error
// IndexRepositories indexes repositories into the persistence layer.
IndexRepositories(ctx context.Context, repos []*claircore.Repository, layer
*claircore.Layer, scnr VersionedScanner) error
// IndexFiles indexes the interesting files into the persistence layer.
IndexFiles(ctx context.Context, files []claircore.File, layer
*claircore.Layer, scnr VersionedScanner) error
// IndexManifest should index the coalesced manifest's content given an
IndexReport.
IndexManifest(ctx context.Context, ir *claircore.IndexReport) error
}
Indexer interface provide the method set required for indexing layer and
manifest contents into a persistent store.
```

46 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Matcher Store
The datastore.MatcherStore interface defines all necessary persistence methods for
Libvuln to provide its functionality.

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface {


Updater
Vulnerability
Enrichment
}
MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface {


Updater
Vulnerability
Enrichment
}
MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface { EnrichmentUpdater

47 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided


// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint
driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string]
[]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string]
[]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities
associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff,
error)
// GC will delete any update operations for an updater which exceeds the
provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency
reasons.
//
// The returned int64 value indicates the remaining number of update operations
needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of
the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with
vulnerabilities at this time

48 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime


time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up
to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime
time.Time) error

} Updater is an interface exporting the necessary methods for updating a vulnerability


database.

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface { Updater Vulnerability Enrichment } MatcherStore aggregates


all interface types

49 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface {


EnrichmentUpdater

// UpdateVulnerabilities creates a new UpdateOperation, inserts the


provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint
driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date
descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string)
(map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string]
[]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities
associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID)
(*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the
provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency
reasons.
//
// The returned int64 value indicates the remaining number of update
operations needing GC.

50 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

// Running this method till the returned value is 0 accomplishes a full GC


of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with
vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime
time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are
up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime
time.Time) error
}
Updater is an interface exporting the necessary methods for updating a
vulnerability database.

package datastore // import "github.com/quay/claircore/datastore"

type EnrichmentUpdater interface { // UpdateEnrichments creates a new


EnrichmentUpdateOperation, inserts the provided // EnrichmentRecord(s), and ensures
enrichments from previous updates are not // queries by clients. UpdateEnrichments(ctx
context.Context, kind string, fingerprint driver.Fingerprint, enrichments
[]driver.EnrichmentRecord) (uuid.UUID, error) } EnrichmentUpdater is an interface exporting
the necessary methods for storing and querying Enrichments.

package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface { Updater Vulnerability Enrichment } MatcherStore aggregates


all interface types

51 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package datastore // import "github.com/quay/claircore/datastore"

type Updater interface {


EnrichmentUpdater

// UpdateVulnerabilities creates a new UpdateOperation, inserts the


provided
// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint
driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date
descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string)
(map[string][]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string]
[]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities
associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID)
(*driver.UpdateDiff, error)
// GC will delete any update operations for an updater which exceeds the
provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency
reasons.
//
// The returned int64 value indicates the remaining number of update
operations needing GC.

52 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

// Running this method till the returned value is 0 accomplishes a full GC


of the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with
vulnerabilities at this time
RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime
time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are
up to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime
time.Time) error
}
Updater is an interface exporting the necessary methods for updating a
vulnerability database.

package datastore // import "github.com/quay/claircore/datastore"

type EnrichmentUpdater interface { // UpdateEnrichments creates a new


EnrichmentUpdateOperation, inserts the provided // EnrichmentRecord(s), and ensures
enrichments from previous updates are not // queries by clients. UpdateEnrichments(ctx
context.Context, kind string, fingerprint driver.Fingerprint, enrichments
[]driver.EnrichmentRecord) (uuid.UUID, error) } EnrichmentUpdater is an interface exporting
the necessary methods for storing and querying Enrichments.

package datastore // import "github.com/quay/claircore/datastore"

type Vulnerability interface {


// get finds the vulnerabilities which match each package provided in the
packages array
// this maybe a one to many relationship. each package is assumed to have
an ID.
// a map of Package.ID => Vulnerabilities is returned.
Get(ctx context.Context, records []*claircore.IndexRecord, opts GetOpts)
(map[string][]*claircore.Vulnerability, error)
}
```
```
package datastore // import "github.com/quay/claircore/datastore"

type MatcherStore interface {


Updater
Vulnerability
Enrichment
}
MatcherStore aggregates all interface types

package datastore // import "github.com/quay/claircore/datastore"

53 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

type Updater interface { EnrichmentUpdater

54 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

// UpdateVulnerabilities creates a new UpdateOperation, inserts the provided


// vulnerabilities, and ensures vulnerabilities from previous updates are
// not queried by clients.
UpdateVulnerabilities(ctx context.Context, updater string, fingerprint
driver.Fingerprint, vulns []*claircore.Vulnerability) (uuid.UUID, error)
// GetUpdateOperations returns a list of UpdateOperations in date descending
// order for the given updaters.
//
// The returned map is keyed by Updater implementation's unique names.
//
// If no updaters are specified, all UpdateOperations are returned.
GetUpdateOperations(context.Context, driver.UpdateKind, ...string) (map[string]
[]driver.UpdateOperation, error)
// GetLatestUpdateRefs reports the latest update reference for every known
// updater.
GetLatestUpdateRefs(context.Context, driver.UpdateKind) (map[string]
[]driver.UpdateOperation, error)
// GetLatestUpdateRef reports the latest update reference of any known
// updater.
GetLatestUpdateRef(context.Context, driver.UpdateKind) (uuid.UUID, error)
// DeleteUpdateOperations removes an UpdateOperation.
// A call to GC must be run after this to garbage collect vulnerabilities
associated
// with the UpdateOperation.
//
// The number of UpdateOperations deleted is returned.
DeleteUpdateOperations(context.Context, ...uuid.UUID) (int64, error)
// GetUpdateOperationDiff reports the UpdateDiff of the two referenced
// Operations.
//
// In diff(1) terms, this is like
//
// diff prev cur
//
GetUpdateDiff(ctx context.Context, prev, cur uuid.UUID) (*driver.UpdateDiff,
error)
// GC will delete any update operations for an updater which exceeds the
provided keep
// value.
//
// Implementations may throttle the GC process for datastore efficiency
reasons.
//
// The returned int64 value indicates the remaining number of update operations
needing GC.
// Running this method till the returned value is 0 accomplishes a full GC of
the vulnstore.
GC(ctx context.Context, keep int) (int64, error)
// Initialized reports whether the vulnstore contains vulnerabilities.
Initialized(context.Context) (bool, error)
// RecordUpdaterStatus records that an updater is up to date with
vulnerabilities at this time

55 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

RecordUpdaterStatus(ctx context.Context, updaterName string, updateTime


time.Time, fingerprint driver.Fingerprint, updaterError error) error
// RecordUpdaterSetStatus records that all updaters from an updater set are up
to date with vulnerabilities at this time
RecordUpdaterSetStatus(ctx context.Context, updaterSet string, updateTime
time.Time) error

} Updater is an interface exporting the necessary methods for updating a vulnerability


database.

package datastore // import "github.com/quay/claircore/datastore"

type EnrichmentUpdater interface {


// UpdateEnrichments creates a new EnrichmentUpdateOperation, inserts the
provided
// EnrichmentRecord(s), and ensures enrichments from previous updates are
not
// queries by clients.
UpdateEnrichments(ctx context.Context, kind string, fingerprint
driver.Fingerprint, enrichments []driver.EnrichmentRecord) (uuid.UUID, error)
}
EnrichmentUpdater is an interface exporting the necessary methods for
storing and querying Enrichments.

package datastore // import "github.com/quay/claircore/datastore"

type Vulnerability interface { // get finds the vulnerabilities which match each package
provided in the packages array // this maybe a one to many relationship. each package is
assumed to have an ID. // a map of Package.ID => Vulnerabilities is returned. Get(ctx
context.Context, records []*claircore.IndexRecord, opts GetOpts) (map[string]
[]*claircore.Vulnerability, error) }

package datastore // import "github.com/quay/claircore/datastore"

type Enrichment interface {


GetEnrichment(ctx context.Context, kind string, tags []string)
([]driver.EnrichmentRecord, error)
}
Enrichment is an interface for querying enrichments from the store.
```

56 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Manifest
A Manifest is analogous to an OCI Image Manifest: it defines the order of layers and how to
retrieve the them.

package claircore // import "github.com/quay/claircore"

type Manifest struct {


// content addressable hash. should be able to be computed via
// the hashes of all included layers
Hash Digest `json:"hash"`
// an array of filesystem layers indexed in the same order as the
cooresponding image
Layers []*Layer `json:"layers"`
}
Manifest represents a docker image. Layers array MUST be indexed in the
order that image layers are stacked.

57 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Matcher
A Matcher performs the heavy lifting of matching manifest contents to relevant
vulnerabilities. These implementations provide the smarts for understanding if a particular
artifact in a layer is vulnerable to a particular advisory in the database.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Matcher interface {


// a unique name for the matcher
Name() string
// Filter informs the Controller if the implemented Matcher is interested
in the provided IndexRecord.
Filter(record *claircore.IndexRecord) bool
// Query informs the Controller how it should match packages with
vulnerabilities.
// All conditions are logical AND'd together.
Query() []MatchConstraint
// Vulnerable informs the Controller if the given package is affected by
the given vulnerability.
// for example checking the "FixedInVersion" field.
Vulnerable(ctx context.Context, record *claircore.IndexRecord, vuln
*claircore.Vulnerability) (bool, error)
}
Matcher is an interface which a Controller uses to query the vulnstore for
vulnerabilities.

The Filter method is used to inform Libvuln the provided artifact is interesting. The
Query method tells Libvuln how to query the security advisory database. The Vulnerable
method reports whether the provided package is vulnerable to the provided vulnerability.
Typically, this would perform a version check between the artifact and the vulnerability in
question.

58 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Package Scanner
A Package Scanner should discover any packages found within the given layer. It is OK for to
discover no packages within a layer.

package indexer // import "github.com/quay/claircore/indexer"

type PackageScanner interface {


VersionedScanner
// Scan performs a package scan on the given layer and returns all
// the found packages
Scan(context.Context, *claircore.Layer) ([]*claircore.Package, error)
}
PackageScanner provides an interface for unique identification or a
PackageScanner and a Scan method for extracting installed packages from an
individual container layer

func NewPackageScannerMock(name, version, kind string) PackageScanner

59 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

RemoteMatcher
RemoteMatcher is an additional interface a Matcher may implement to skip the database for
matching results and use an external API.

package driver // import "github.com/quay/claircore/libvuln/driver"

type RemoteMatcher interface {


QueryRemoteMatcher(ctx context.Context, records []*claircore.IndexRecord)
(map[string][]*claircore.Vulnerability, error)
}
RemoteMatcher is an additional interface that a Matcher can implement.

When called the interface can invoke the remote matcher using an HTTP API
to
fetch new vulnerabilities associated with the given IndexRecords.

The information retrieved from this interface won't be persisted into the
claircore database.

60 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Repository Scanner
A RepositoryScanner should identify any repositories discovered in the provided layer. It is
OK for the scanner to identify no repositories.

package indexer // import "github.com/quay/claircore/indexer"

type RepositoryScanner interface {


VersionedScanner
Scan(context.Context, *claircore.Layer) ([]*claircore.Repository, error)
}

61 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Resolver
A Resolver is used to analyze and modify the post-coalesced index report. This is useful for
operations that need all context from an index report.

package indexer // import "github.com/quay/claircore/indexer"

type Resolver interface {


Resolve(context.Context, *claircore.IndexReport, []*claircore.Layer)
*claircore.IndexReport
}
Resolver is used for any reasoning that needs to be done with all the
layers
in context.

Resolvers are called at the end of the coalesce step when reports from
separate scanners are merged.

Any Resolvers' Resolve() methods are called (in no set order) at the end of the coalesce
step after reports from separate scanners are merged.

62 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

RPCScanner
RPCScanner is an optional interface a Scanner may implement. When implemented, the
scanner's Configure method will be called with a ConfigDeserializer function and an
HTTP client. The Scanner may pass its config as an argument to the ConfigDeserializer
function to populate the struct and use the HTTP client for any remote access necessary
during the scanning process.

package indexer // import "github.com/quay/claircore/indexer"

type RPCScanner interface {


Configure(context.Context, ConfigDeserializer, *http.Client) error
}
RPCScanner is an interface scanners can implement to receive configuration
and denote that they expect to be able to talk to the network at run time.

package indexer // import "github.com/quay/claircore/indexer"

type RPCScanner interface {


Configure(context.Context, ConfigDeserializer, *http.Client) error
}
RPCScanner is an interface scanners can implement to receive configuration
and denote that they expect to be able to talk to the network at run time.

package indexer // import "github.com/quay/claircore/indexer"

type ConfigDeserializer func(interface{}) error ConfigDeserializer can be thought of as an


Unmarshal function with the byte slice provided.

This will typically be something like (*json.Decoder).Decode.

63 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Updater
An Updater is responsible for performing run-time fetching and parsing of a security
database. The returned vulnerabilities will be written to claircore's database and used in
vulnerability matching.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {


Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {


Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)


} Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface

64 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

combining the method set of a Fetcher and a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface {


Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)
}
Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which
uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the
contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should
assume contents are uncompressed and ready for parsing. Parse(ctx context.Context,
contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is
embedded into the Updater interface.

Parse should be called with an io.ReadCloser struct where the contents


of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface { Name() string Fetcher Parser } Updater is an aggregate interface
combining the method set of a Fetcher and a Parser and forces a Name() to be provided

65 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface {


Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)
}
Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which
uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface { // Parse should take an io.ReadCloser, read the contents, parse the
contents // into a list of claircore.Vulnerability structs and then return // the list. Parse should
assume contents are uncompressed and ready for parsing. Parse(ctx context.Context,
contents io.ReadCloser) ([]*claircore.Vulnerability, error) } Parser is an interface which is
embedded into the Updater interface.

Parse should be called with an io.ReadCloser struct where the contents


of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fingerprint string


Fingerprint is some identifying information about a vulnerability database.
```
```
package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {


Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

66 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)


} Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface {


// Parse should take an io.ReadCloser, read the contents, parse the
contents
// into a list of claircore.Vulnerability structs and then return
// the list. Parse should assume contents are uncompressed and ready for
parsing.
Parse(ctx context.Context, contents io.ReadCloser)
([]*claircore.Vulnerability, error)
}
Parser is an interface which is embedded into the Updater interface.

Parse should be called with an io.ReadCloser struct where the contents


of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fingerprint string Fingerprint is some identifying information about a vulnerability


database.

67 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package driver // import "github.com/quay/claircore/libvuln/driver"

type Configurable interface {


Configure(context.Context, ConfigUnmarshaler, *http.Client) error
}
Configurable is an interface that Updaters can implement to opt-in to
having
their configuration provided dynamically.
```
```
package driver // import "github.com/quay/claircore/libvuln/driver"

type Updater interface {


Name() string
Fetcher
Parser
}
Updater is an aggregate interface combining the method set of a Fetcher and
a Parser and forces a Name() to be provided

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fetcher interface { Fetch(context.Context, Fingerprint) (io.ReadCloser, Fingerprint, error)


} Fetcher is an interface which is embedded into the Updater interface.

When called the interface should determine if new security advisory data
is available. Fingerprint may be passed into in order for the Fetcher to
determine if the contents has changed

If there is new content Fetcher should return a io.ReadCloser where the new
content can be read. Optionally a fingerprint can be returned which uniquely
identifies the new content.

If the conent has not change an Unchanged error should be returned.

68 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

package driver // import "github.com/quay/claircore/libvuln/driver"

type Parser interface {


// Parse should take an io.ReadCloser, read the contents, parse the
contents
// into a list of claircore.Vulnerability structs and then return
// the list. Parse should assume contents are uncompressed and ready for
parsing.
Parse(ctx context.Context, contents io.ReadCloser)
([]*claircore.Vulnerability, error)
}
Parser is an interface which is embedded into the Updater interface.

Parse should be called with an io.ReadCloser struct where the contents


of a security advisory database can be read and parsed into an array of
*claircore.Vulnerability

package driver // import "github.com/quay/claircore/libvuln/driver"

type Fingerprint string Fingerprint is some identifying information about a vulnerability


database.

package driver // import "github.com/quay/claircore/libvuln/driver"

type Configurable interface {


Configure(context.Context, ConfigUnmarshaler, *http.Client) error
}
Configurable is an interface that Updaters can implement to opt-in to
having
their configuration provided dynamically.

package driver // import "github.com/quay/claircore/libvuln/driver"

type ConfigUnmarshaler func(interface{}) error ConfigUnmarshaler can be thought of as an


Unmarshal function with the byte slice provided, or a Decode function.

The function should populate a passed struct with any configuration


information.

69 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

UpdaterSetFactory
An UpdaterSetFactory is a factory for runtime construction and configuration for Updaters.

package driver // import "github.com/quay/claircore/libvuln/driver"

type UpdaterSetFactory interface {


UpdaterSet(context.Context) (UpdaterSet, error)
}
UpdaterSetFactory is used to construct updaters at run-time.

func StaticSet(s UpdaterSet) UpdaterSetFactory

package driver // import "github.com/quay/claircore/libvuln/driver"

type UpdaterSetFactory interface {


UpdaterSet(context.Context) (UpdaterSet, error)
}
UpdaterSetFactory is used to construct updaters at run-time.

func StaticSet(s UpdaterSet) UpdaterSetFactory

package driver // import "github.com/quay/claircore/libvuln/driver"

type UpdaterSet struct { // Has unexported fields. } UpdaterSet holds a deduplicated set of
updaters.

func NewUpdaterSet() UpdaterSet func (s *UpdaterSet) Add(u Updater) error func (s


*UpdaterSet) Merge(set UpdaterSet) error func (s *UpdaterSet) RegexFilter(regex string)
error func (s *UpdaterSet) Updaters() []Updater

70 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

VersionFilter
VersionFilter is an additional interface a Matcher may implement. If implemented,
Libvuln will attempt to use the database and the normalized version field of a package to
filter vulnerabilities in the database. This is an opt-in optimization for when a package
manager's version scheme can be normalized into a claircore.Version .

package driver // import "github.com/quay/claircore/libvuln/driver"

type VersionFilter interface {


VersionFilter()
// VersionAuthoritative reports whether the Matcher trusts the database-
side
// filtering to be authoritative.
//
// A Matcher may return false if it's using a versioning scheme that can't
// be completely normalized into a claircore.Version.
VersionAuthoritative() bool
}
VersionFilter is an additional interface that a Matcher can implement to
opt-in to using normalized version information in database queries.

71 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Versioned Scanner
A versioned scanner is typically embedded into other scanner types. It drives claircore's
ability to register and understand when updaters have been changed. Functions that want
to work with a generic scanner type should use a VersionedScanner .

Implementers of this interface must provide a unique name. Making changes to a scanner's
implementation must return a new value from Version . Implementers must return the
correct kind: one of "package", "distribution", or "repository"

package indexer // import "github.com/quay/claircore/indexer"

type VersionedScanner interface {


// unique name of the distribution scanner.
Name() string
// version of this scanner. this information will be persisted with the
scan.
Version() string
// the kind of scanner. currently only package is implemented
Kind() string
}
VersionedScanner can be embedded into specific scanner types. This allows
for methods and functions which only need to compare names and versions of
scanners not to require each scanner type as an argument.

72 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Vulnerability Report
A Vulnerability Report is a structure describing a specific manifest, its contents, and the
vulnerabilities affecting its contents.

package claircore // import "github.com/quay/claircore"

type VulnerabilityReport struct {


// the manifest hash this vulnerability report is describing
Hash Digest `json:"manifest_hash"`
// all discovered packages in this manifest keyed by package id
Packages map[string]*Package `json:"packages"`
// all discovered distributions in this manifest keyed by distribution id
Distributions map[string]*Distribution `json:"distributions"`
// all discovered repositories in this manifest keyed by repository id
Repositories map[string]*Repository `json:"repository"`
// a list of environment details a package was discovered in keyed by
package id
Environments map[string][]*Environment `json:"environments"`
// all discovered vulnerabilities affecting this manifest
Vulnerabilities map[string]*Vulnerability `json:"vulnerabilities"`
// a lookup table associating package ids with 1 or more vulnerability ids.
keyed by package id
PackageVulnerabilities map[string][]string `json:"package_vulnerabilities"`
// a map of enrichments keyed by a type.
Enrichments map[string][]json.RawMessage `json:"enrichments"`
}
VulnerabilityReport provides a report of packages and their associated
vulnerabilities.

A Vulnerability Report is package focused.

Unpacking a report is done by mapping the keys in the PackageVulnerabilities field to the
data structures in other lookup maps.

For example:

for pkgID, vulnIDS := range report.PackageVulnerabilities {


// get package data structure
pkg := report.Packages[pkgID]

for _, vulnID := range vulnIDS {


vuln := report.Vulnerabilities[vulnID]
fmt.Printf("package %+v affected by vuln %+v", pkg, vuln)
}
}

73 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Contributors
These topics cover helpful tips for contributing to Claircore.

• Local Development
• Logging
• Misc
• Tests

74 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

How to contribute
The preferred workflow is to fork the quay/claircore repository, push a feature branch to
the new fork, then open a pull request. All pull requests should be targeted to the main
branch outside of exceptional circumstances.

Testing
As many tests as possible should run with the standard go test invocations. Adding the
special tag integration (e.g. go test -tags integration ./... ) will also run "integration"
tests. The project interprets "integration" tests to mean any test that would need external
resources, such as:

• External web servers


• External network access
• Out-of-process databases
• Large test fixtures

After at least one run with the integration tag, the tests should cache needed resources
and run as many tests as possible. See also the test/integration package.

Pull Requests
The Pull Request (PR) is the unit of code review. Claircore's review flow treats a feature
branch as a stack of patches to be applied. That is to say, the feature branch should be
rebased onto the target branch and have well-organized commits. Merge commits are
disallowed. If the author would prefer to not rewrite commit history while working through
reviews, fixup commits are the suggested way to achieve that. As many requirements as
possible are enforced by CI, like:

• Commits being signed off


• Commit messages having a properly formed subject
• Go modules being tidied

Please use the "draft" option if the branch is not ready. Please enable the "allow edits by
maintainers" option.

75 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

The maintainers may rebase, push, and merge contributors' branches. This may necessitate
doing a git reset <remote>/<branch> to update a local branch.

Conventions
Git commits should be formatted like "subject: summary" and avoid going over 80
characters per line. The "subject" is usually the package affected by the commit (like jar or
rhel -- the relative path isn't needed) but sometimes a broader category (like docs , all ,
or cicd ) is OK.

All the helper scripts should handle the "normal" convention ( origin is quay/claircore
and fork is one's personal fork) and the "British" convention ( origin is one's personal fork
and upstream is quay/claircore ).

More detailed contributor documentation can be found in the project documentation.

76 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

The claircore project has switched from a git log based changelog to a git notes based
changelog.

This has the benefit of making the changelog more human-friendly, as it can have prose
describing changes now, but makes adding entries a bit more involved. A full understanding
of git notes is helpful for working with the changelog, but not required. If the reader has
worked with the notes feature before, the changelog entries are stored under the
changelog ref. For other users, there are some helper scripts in .github/scripts .

Basics of git notes

Git notes is a mechanism for storing additional information alongside commits without
modifying the commits. It does this by creating a ref full of files named after the commits,
with their contents being the notes. This scheme requires some special care and tooling --
see the documentation for more information.

Helper scripts

The primary helper script is changelog-edit . It allows a user to sync down notes, edit an
entry, or both. See the output of the h flag for more information.

The other script of interest is changelog-render , which can be used to render out the
changelog on demand, assuming the changelog notes have been pulled locally.

The changelog-update script uses changelog-render to add to the CHANGELOG.md file in


the repository root.

Formatting

Broadly, changelog entries should be formatted like commit messages without any trailers.
Entries are turned into list items, with the subject being the bullet point and the body of the
entry being the "body" of the item, or hidden behind details elements when using HTML-
enabled output.

The entries are almost always rendered as markdown, so using minimal markdown is OK.
Anything requiring excessive markdown is probably better served as documentation proper,
rather than a changelog entry.

77 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Local Development
A local development environment is implemented via docker-compose.

Usage
Several make targets are defined for working with the local development environment.

claircore-db-up - creates just the claircore database useful for running


integration tests without test servers
claircore-db-restart - destroys and recreates a fresh database. localhost:5434

Tests
Several make targets are defined for working with tests.

integration - run the integration test suite. requires the claircore-db to be


up. run `make clair-db-up` before this target
unit - run the unit test suite.
bench - runs the benchmarks
integration-v - runs the integration test suite with verbose
unit-v - runs the unit test suite with verbose

78 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Logging
All the logging in claircore is done with zerolog via context.Context values. The zlog
package takes OpenTelemetry labels and attaches them to zerolog events.

This allows for claircore's logging to be used consistently throughout all the packages
without having unintended prints to stderr.

How to Log

Adding Context

In a function, use zlog to add key-value pairs of any relevant context:

ctx = zlog.ContextWithValues(ctx,
"component", "Example.Logger")

Alternatively, the go.opentelemetry.io/otel/baggage package can be used for more


explicit control around the baggage values.

Logging style

Constant Messages

Zerolog emits lines when the Msg or Msgf methods are called. Project style is to not use
Msgf . Any variable data should be set as key-value pairs on the Event object.

For example, don't do this:

zlog.Info(ctx).Msgf("done at: %v", time.Now())

Do this instead:

zlog.Info(ctx).
Time("time", time.Now()).
Msgf("done")

79 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Grammar

When noting the change during a chunk of work, make sure that the log messages scan as
visually similar. Usually, this means formatting messages into "${process} ${event}". For
example:

frob start
frob initialized
frob ready
frob success
frob done

Is much easier to scan than:

starting to frob
initialized frobber
ready for frobbing
did frob
done with frobing

Don't log and return

When handling an error, code should only log it if it does not propagate it. The code that
ultimately handles the error is responsible for deciding what to do with it. Logging and
returning ends up with the same message repeated multiple times in the logs.

Levels

Claircore attempts to have consistent leveled logging. The rules for figuring out what level to
use is:

• Panic

There's some occurrence that means the process won't work correctly.

• Fatal

Unused, because it prevents defers from running.

• Error

Something unexpected occurred and the process can continue, but a human needs to
be notified. An error will be returned for this request.

80 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

• Warn

Something unexpected occurred and the process can continue. An error will be
returned for this request.

• Info

Some information that may be useful to an operator. Examples include a timer-based


process starting and ending, a user request starting and ending, or a summary of work
done.

• Debug

Some information that may be useful to a developer. Examples include entering and
exiting functions, stepping through a function, or specific file paths used while work is
being done.

81 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Here's various codebase conventions that don't have dedicated pages:

• URLs URLs in code should be annotated with a //doc:url directive comment. See the
the internal/cmd/mdbook-injecturls command for documentation on how the
preprocessor works. The list of keywords isn't an allowlist, so an invocation like the
following should list the ones actually used in the documentation using a command
like git grep injecturls -- :/docs/*.md .

82 of 83 11/30/2023, 5:26 PM
Claircore Documentation https://quay.github.io/claircore/print.html

Tests
Tests in the claircore module may use various helpers underneath the test directory. Using
these packages outside of testing code is disallowed. Assert packages are disallowed; the
go-cmp package is the only external package helper allowed.

Tests that use external resources or generate test fixtures should be annotated according to
the integration package.

Caching
Tests using the integration package cache generated and downloaded assets into a
directory named clair-testing inside the directory reported by os.UserCacheDir . For
example, on a Linux system, the cache directory will be (in sh notation)
${XDG_CACHE_HOME-$HOME/.cache}/clair-testing .

83 of 83 11/30/2023, 5:26 PM

You might also like