Turbinia Local and Manual Installation Instructions

Introduction

Turbinia can be run on the Google Cloud Platform, on local machines, or in a hybrid mode. See the “how it works” documentation for more details on what the architecture looks like for each of these installation types. This page covers the local installation as well as the manual steps for hybrid and cloud installation. If you are setting up a GCP cloud or hybrid installation, it is highly recommended and much simpler to use the terraform installation method to bootstrap those environments.

Each section of this document indicates which installation types it applies to (cloud, hybrid or local), so you only need to follow the relevant steps for your installation type.

Prerequisites

Turbinia requires all worker nodes to have equal access to all Evidence data. For Google Cloud this means using Google Cloud Storage (GCS), but for local and hybrid configurations the easiest setup is to have NFS or a SAN mounted on a common path on each worker. All output should also be written to this common directory so that when new Evidence is generated by a Task, that the other worker nodes can access it for processing. Turbinia can also write output to GCS even when running locally (see the GCS_OUTPUT_PATH variable in the config).

Installation

To run Turbinia it’s recommended that you have at least two machines or cloud instances, one for the server and one or more for workers. In a small or development setup, you can also run both the server and worker on a single instance.

GCP Setup

This section is required for cloud and hybrid configurations.

  • Create or select a Google Cloud Platform project in the Google Cloud Console.
  • Determine which GCP zone and region that you wish to deploy Turbinia into. Note that one of the GCP dependencies is Cloud Functions, and that only works in certain regions, so you will need to deploy in one of the supported regions.
  • Enable Cloud Functions.
  • Follow the instructions to:
    • Enable Cloud Pub/Sub
    • Create a new Pub/Sub topic and subscription (pull type with 600s timeout). These can use the same base name (the part after topics/ and subscription/ in the paths).
    • Please take a note of the topic name for the configuration steps, as this is what you will set the PUBSUB_TOPIC config variable to.
  • Enable Cloud Datastore
    • Go to Datastore in the cloud console
    • Hit the Create Entity button
    • Select the same region that you selected in the previous steps. No need to create any Entities after selecting your region

Create a GCE Instance for the Server

This section is required for cloud configurations.

  • Create a new GCE instance from a recent version of Debian or Ubuntu (currently 18.0.4 is recommended). Once the host is configured we’ll later clone the disk to use for the workers.
    • This should work on other Linux flavors, but these are untested, and the installation steps would need to be adapted.

Create a Google Cloud Storage (GCS) Bucket

This section is required for cloud installations.

  • Create a new GCS bucket and take note of the bucket name as this will be referenced later by the GCS_OUTPUT_PATH variable.

Core Installation

This section is required for cloud, hybrid and local configurations.

Preparation

All steps listed below must be completed on all servers/workers unless otherwise noted. If you are installing in a cloud environment, you will only have a server VM at this point, and we will later clone this instance to create the workers.

  • Install dependencies
    • sudo apt-get install python-dev build-essential python-setuptools python-pip python-virtualenv liblzma-dev git john
  • Create a turbinia user with password-less sudo access
    • sudo adduser --disabled-password turbinia
    • echo "turbinia ALL = (root) NOPASSWD: /bin/mount,/bin/umount,/sbin/losetup" | sudo tee -a /etc/sudoers.d/turbinia
  • Prepare configuration directory:
    • sudo mkdir /etc/turbinia
    • sudo chown turbinia /etc/turbinia
  • Checkout git at release branch for config and other setup
    • git clone https://github.com/google/turbinia.git
    • cd turbinia
    • git branch -l | grep release
    • git checkout <latest release branch from previous step>

Installation and Configuration

  • Log in as turbinia
    • sudo su - turbinia
  • Install Turbinia
    • Note: You may want to install this into a virtual-environment with venv or pipenv to reduce potential dependency conflicts and isolate these packages into their own environment.
    • pip3 install turbinia for the server
    • pip3 install turbinia[worker] for the worker
    • pip3 install turbinia[dev] if you want to run tests or get the development dependencies.
    • If you are running a local installation:
      • pip3 install turbinia[local]
  • Install Worker binary dependencies
    • You can install Plaso from the GIFT PPA, or see here for other packaged installations.
    • There are a few other binary dependencies that are not packaged with Ubuntu or PyPi, so these will need to be installed manually: bulk_extractor, hindsight (this one is technically in PyPi, but since it’s not Python3 yet it needs to be installed separately) and Volatility. Alternately you can disable the Jobs that have those dependencies. You can do this by setting the DISABLED_JOBS in the /etc/turbinia/turbinia.conf config after it is installed (see below). You can see the list of Jobs with turbiniactl listjobs after everything is set up.
  • Create and configure the Turbinia configuration file.
    • cp <git clone path>/turbinia/config/turbinia_config_tmpl.py /etc/turbinia/turbinia.conf
    • Alternately, you can either put the file in /home/$USER/.turbiniarc or in another directory and then point the TURBINIA_CONFIG_PATH environment variable to that directory.
    • Edit the config file to match your local installation details.
      • Match the PUBSUB_TOPIC variable in the configuration to the name of the topic you created in GCP.
      • If you are running Turbinia locally, make sure to set GCS_OUTPUT_PATH to None.
      • For local and hybrid installations:
        • Set SHARED_FILESYSTEM = True
      • For local installations:
        • Set STATE_MANAGER = 'Redis'
        • Set TASK_MANAGER = 'Celery'
        • Configure the CELERY*, KOMBU* and REDIS* variables as appropriate for your config.
      • Set the following
    • Configure the OUTPUT_DIR, TMP_DIR, and MOUNT_DIR_PREFIX to match your local system. On the worker nodes, create the corresponding directories and make sure they are owned by the turbinia user.
  • Configure the init scripts to run Turbinia on start
    • cp <git clone path>/turbinia/tools/turbinia@.service /etc/systemd/system/turbinia@server for the server
    • cp <git clone path>/turbinia/tools/turbinia@.service /etc/systemd/system/turbinia@psqworker for a GCP worker
    • cp <git clone path>/turbinia/tools/turbinia@.service /etc/systemd/system/turbinia@celeryworker for a local (non-cloud) installation.
    • Follow the instructions at the top of the turbinia/tools/turbinia@.service file to enable these services.

GCP Installation

This section is required for cloud and hybrid configurations.

  • Install google-cloud-sdk
  • Create a scoped service account (this is the best option) with the following roles:
    • Cloud Datastore User: Used by PSQ to store result data, and in the future by the Task Manager to store queryable task data
    • Pub/Sub Editor: Used by clients to talk to Turbinia, and by the Task Manager to talk to workers
    • Storage Object Admin and Storage Legacy Bucket Reader: Only required on the GCS bucket used by Turbinia, if any. See GCP Turbinia for details
    • Compute Instance Admin: Used to list instances and to attach disks to instances
    • Service Account User: Used when attaching disks
    • Cloud Functions Developer: Used by turbiniactl to query task status
  • Create a new key for your service account, and save it on server/workers and then configure init scripts in /etc/systemd/system/turbinia* to point to it by setting the GOOGLE_APPLICATION_CREDENTIALS var similar to ExecStartPre=+/bin/sh -c '/bin/systemctl set-environment GOOGLE_APPLICATION_CREDENTIALS="/home/turbinia/turbinia-service-account-creds.json"'
  • Add the service account to the gcloud auth
    • gcloud auth list
    • gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
  • Alternately you can run Turbinia under your own credentials (NOT RECOMMENDED except for development environments)
    • Run gcloud auth login (may require you to copy/paste url to browser)
    • Run gcloud auth application-default login

Create GCP workers

This section is required for cloud configurations.

  • Stop the server instance that has been configured above.
  • Create a new image from the server VM’s disk.
  • Create a new Instance Template using the newly created image.
  • Create a new Managed Instance Group from the newly created Instance Template.

Deploy Cloud Functions

This section is required for cloud and hybrid configurations.

  • cd <git clone path>/tools/gcf_init && ./deploy_gcf.py

Local Turbinia

Running Turbinia locally using Docker. This setup does not require Google Cloud Platform.

See here for detailed instructions.