BayesianPlugin

A FederationLib plugin that classifies scanned content with BayesianServer and trains it with the classifications assigned to evidence. It provides FederationLib's Bayesian classification and training, which used to be built into FederationLib.

Table of contents

• BayesianPlugin
  • Table of contents
  • What it does
  • Configuration
  • Installation
    • Docker
  • Migrating from FederationLib's built-in implementation
  • Building and testing
    • Test environment
    • Running the tests
• License

What it does

The plugin does two things: it classifies the content FederationLib scans, and it learns from the classifications operators give to evidence.

Classifying content. When content is scanned with POST /scan, every evidence item that has text is sent to BayesianServer (in one request) and classified as NORMAL, SUSPICIOUS or MALICIOUS, with NORMAL as the negative label. Content classified as NORMAL is learned as such later if it isn't classified in the meantime (implicit_learning, with BayesianServer's --implicit-delay), so the model sees ordinary traffic too. FederationLib takes it from there: the classification appears in the scan's classification, counts towards the risk score through the CLASSIFICATION_* scanning rules, and describes the evidence of reports that are generated automatically from a scan. The scan request's top_k and threshold parameters are passed on to BayesianServer.

Learning from classifications. Whenever evidence is given a classification, its text is sent to BayesianServer to train the model. This happens when:

• evidence is submitted with a classification,
• evidence is classified (or reclassified) with PATCH /evidence/{uuid}/classify,
• a report is closed with a classification.

The text is sent as a correction (PATCH / of BayesianServer 1.0.4 and later): whatever the text was learned with before is replaced, so the latest classification of a text is the one the model knows. With older servers, or --corrections false, it's learned with PUSH / instead. Learning can be turned off with the learning option.

Classifications BayesianServer missed (eg; while it was restarting) are learned by federationlib bayesian-sync, which the docker image runs every hour, and the content of deleted evidence is forgotten (unless other classified evidence has the same text).

When content isn't classified. The plugin would rather not classify content than guess. It skips the classification when:

• the model hasn't been trained enough yet, it needs at least minimum_documents documents in total and for each of NORMAL, SUSPICIOUS and MALICIOUS,
• the model contains a label the plugin doesn't know,
• most of the content's words are unknown to a Naive Bayes model (classify_known_tokens, the logistic model also scores the characters of unknown words),
• the top label is SUSPICIOUS or MALICIOUS but BayesianServer doesn't predict it (it is below the server's decision threshold, which the logistic model calibrates for a false positive rate),
• the top label is SUSPICIOUS or MALICIOUS and the probability that the content isn't NORMAL is below minimum_confidence. A single classification can block content, report its author and hold back their reputation, so a spam classification needs to be confident.

Telling SUSPICIOUS and MALICIOUS apart. The text alone tells them apart poorly, what makes spam MALICIOUS is that it points somewhere. A contact route in the content (a link, a Telegram link, a handle or a phone number, or a named entity FederationLib resolved that isn't whitelisted), or an author that is blacklisted or has a negative reputation, confirms MALICIOUS. With spam_severity upgrade (default) the model decides and confirmed SUSPICIOUS spam becomes MALICIOUS; with confirm confirmed spam is MALICIOUS and other spam SUSPICIOUS (the label policy); with model the model decides alone.

The confidence of a NORMAL classification is the probability BayesianServer gives it, the confidence of a SUSPICIOUS or MALICIOUS classification is the probability that the content isn't NORMAL (1 - P(NORMAL)).

If BayesianServer is unavailable, scans and other operations still succeed, the content just isn't classified and the error is logged.

Direct access to BayesianServer. When the proxy option is enabled, the root operator can use BayesianServer's API through FederationLib, every request below /bayesian is passed on to BayesianServer as-is and its response is returned unchanged:

FederationLib	BayesianServer
GET /bayesian/	GET / (status)
POST /bayesian/	POST / (classify)
PUSH /bayesian/	PUSH / (train)
GET /bayesian/health	GET /health
POST /bayesian/analytics	POST /analytics

The request's method, query string, body and Content-Type are forwarded, BayesianServer's status code, Content-Type and body are returned. Any other operator is refused (403), and anonymous requests need to authenticate (401). The operator's access token is never sent to BayesianServer.

The proxy is disabled by default, since it gives direct control over the model, including training it without classified evidence. While disabled, /bayesian responds like a path that doesn't exist.

Handler	Handles
ContentScanHandler	CONTENT_SCAN, classifies scanned content
EvidenceClassifiedHandler	RECORD_CHANGE (EVIDENCE_CLASSIFIED), trains the model
EvidenceDeletedHandler	RECORD_CHANGE (EVIDENCE_DELETED), forgets the content
BayesianProxyHandler	The /bayesian/* route, proxies BayesianServer's API

Configuration

The plugin has its own ConfigLib configuration named bayesian_plugin, every value can also be set with its environment variable. The defaults connect to the BayesianServer bundled with the FederationLib docker image.

Name	Environment Variable	Type	Default Value	Description
host	BAYESIAN_PLUGIN_HOST	string	127.0.0.1	The host of BayesianServer
port	BAYESIAN_PLUGIN_PORT	integer	6380	The port of BayesianServer
ssl	BAYESIAN_PLUGIN_SSL	boolean	false	Connect to BayesianServer with HTTPS
classify_known_tokens	BAYESIAN_PLUGIN_CLASSIFY_KNOWN_TOKENS	boolean	true	Skip the classification when the majority of the content's tokens are unknown
minimum_documents	BAYESIAN_PLUGIN_MINIMUM_DOCUMENTS	integer	10	Training documents required in total and per label before content is classified
minimum_confidence	BAYESIAN_PLUGIN_MINIMUM_CONFIDENCE	float	0.9	Minimum probability that content isn't NORMAL for a spam classification
learning	BAYESIAN_PLUGIN_LEARNING	boolean	true	Train BayesianServer with the text content of classified evidence
spam_severity	BAYESIAN_PLUGIN_SPAM_SEVERITY	string	upgrade	How spam is split into SUSPICIOUS/MALICIOUS: upgrade, confirm or model
implicit_learning	BAYESIAN_PLUGIN_IMPLICIT_LEARNING	boolean	true	Ask BayesianServer to learn content classified as NORMAL later
proxy	BAYESIAN_PLUGIN_PROXY	boolean	false	Proxy BayesianServer's API below /bayesian for the root operator

Installation

The plugin is loaded by FederationLib through the plugins configuration (FEDERATION_PLUGINS) using its package name, net.nosial.bayesian_plugin. FederationLib (and ConfigLib and LogLib2) are provided by FederationLib at runtime.

make target/release/net.nosial.bayesian_plugin.ncc
ncc install --package="$PWD/target/release/net.nosial.bayesian_plugin.ncc" --yes --reinstall

Docker

The FederationLib docker image installs plugins listed in REQUIRE_PLUGINS before it initializes:

environment:
  - REQUIRE_PLUGINS=/opt/plugins/net.nosial.bayesian_plugin.ncc   # Anything `ncc install` accepts
  - FEDERATION_PLUGINS=net.nosial.bayesian_plugin

Migrating from FederationLib's built-in implementation

No migration is needed, FederationLib includes this plugin as one of the default plugins, the plugin uses the same BayesianServer and model, nothing needs to be retrained.

This plugin does not conflict with any other plugin by default.

Building and testing

make                  # Builds target/release and target/debug
make test-env         # Starts the test environment (FederationLib with the plugin installed, see below)
make test             # Runs the tests (phpunit) against the test environment, requires the release build
make test-env-down    # Removes the test environment and its data

Test environment

The tests run against a live FederationLib server with the plugin installed, started with Docker Compose (docker-compose.yml):

Service	Description	Port
app	FederationLib's published dev image (ghcr.io/nosial/federationlib:dev) with the plugin installed	7000 (FEDERATION_PORT)
mariadb	FederationLib's database	-
redis	FederationLib's cache	-

The Dockerfile builds the plugin from source and adds it to FederationLib's image. FederationLib's entrypoint installs it and enables it through REQUIRE_PLUGINS and FEDERATION_PLUGINS, with its proxy option enabled. The BayesianServer bundled with FederationLib's image isn't published, the tests reach it through the plugin's /bayesian proxy as the root operator. The image is only used for testing and is never published.

• make test-env builds with --pull, so the tests always run against the latest build of FederationLib's dev image rather than an outdated local copy. It then waits until FederationLib and BayesianServer (through the proxy) respond.
• The plugin inside the container is the one built when the image was built, not target/release. Run make test-env again after changing the plugin, otherwise the tests against the server use the old plugin.
• Nothing is persisted: make test-env-down removes the database and the trained model.

Running the tests

The tests import FederationLib from the installed net.nosial.federation package. They're run against FederationLib's dev branch, so the plugin is always tested against the latest working build of the server. Install FederationLib from a checkout of that branch (its dependencies are installed with it), then the plugin package:

git clone --branch dev https://github.com/nosial/federationlib
(cd federationlib && ncc build --configuration release && ncc install --package="$PWD/target/release/net.nosial.federation.ncc" --yes --reinstall)
make target/release/net.nosial.bayesian_plugin.ncc
ncc install --package="$PWD/target/release/net.nosial.bayesian_plugin.ncc" --yes --reinstall

The tests are configured by phpunit.xml, an environment variable that is already set takes precedence:

Environment Variable	Default	Description
SERVER_ENDPOINT	http://172.17.0.1:7000	The FederationLib server of the test environment
SERVER_ACCESS_TOKEN	abcdefghijklmnopqrstuvwxyz123456	The root operator's access token (FEDERATION_ACCESS_TOKEN), for /bayesian
NCC_BUILD_OUTPUT_PATH	target/release/...	The plugin .ncc package to import

The endpoints use the Docker host's bridge address (172.17.0.1), the same as FederationLib's tests, so they work from the host and from a CI job's container. Where the bridge isn't reachable (eg; Docker Desktop), use http://127.0.0.1:7000. The tests train the BayesianServer, so only run them against the test environment. A test fails rather than being skipped if the server is unreachable.

License

This project is licensed under the MIT License - see the LICENSE file for details.