Developing

Overview

ThreatIngestor can be easily extended to support other input and output mechanisms through Source (input) and Operator (output) Python plugins.

Source Plugins

To add support for a new source, simply create a new Python file in the sources folder (e.g. examplesource.py) and extend the sources.Source class with a class called Plugin, overwriting both the __init__ and run methods:

from threatingestor.sources import Source

class Plugin(Source):

    def __init__(self, name, my_required_arg):
        """Args should be url, auth, etc, whatever is needed to set up the object.

        The name argument is required for all Source plugins, and is used internally.
        """
        self.name = name
        self.my_required_arg = my_required_arg

    def run(self, saved_state):
        """Run and return (saved_state, list(Artifact))"""
        artifact_list = []

        return saved_state, artifact_list

You will most likely want to use the sources.Source.process_element method to build the artifacts list. Check inline documentation, and see sources/twitter.py and sources/rss.py for examples.

Any arguments specified in __init__ can be passed in from correspondingly named keys in the config.yml section at runtime:

- name: myexample
  module: examplesource
  my_required_arg: Some value

Operator Plugins

Once artifacts are collected by a source plugin, they’re sent to any configured operator plugins for processing or export. Adding an operator plugin is much the same as adding a source. Create a Python file in the operators folder and extend the operators.Operator class, with a class named Plugin, overwriting the __init__ and handle_artifact methods:

import threatingestor.artifacts
from threatingestor.operators import Operator

class Plugin(Operator):

    def __init__(self, my_required_arg, artifact_types=None, filter_string=None, allowed_sources=None):
        """Args should be url, auth, etc, whatever is needed to set up the object.

        Set self.artifact_types to a list of Artifacts supported by the plugin.
        """
        super(ExampleOperator, self).__init__(artifact_types, filter_string, allowed_sources)
        self.artifact_types = artifact_types or [artifacts.IPAddress, artifacts.Domain]

    def handle_artifact(self, artifact):
        """Override with the same signature"""
        # process artifact

Operators will only be run on artifacts in their artifact_types list.

As with source modules, any arguments specified in __init__ can be passed in from correspondingly named keys in the config.yml section at runtime:

- name: myexample
  module: exampleoperator
  my_required_arg: Some value