Bump VERSION file

#105 Change default api page size to 1000

.github/issue_template.md

@@ -9,6 +9,12 @@ Before raising an issue here, answer the following questions for yourself, pleas
   (Otherwise ask on the Netbox mailing list, please: https://groups.google.com/d/forum/netbox-discuss)
 * Have you looked through the issues already resolved?
 
+Please try this means to get help before opening an issue here:
+
+* On the networktocode Slack in the #netbox-docker channel: http://slack.networktocode.com/
+* On the networktocode Slack in the #netbox channel: http://slack.networktocode.com/
+* On the Netbox mailing list: https://groups.google.com/d/forum/netbox-discuss
+
 -->
 
 ## Current Behavior
@@ -30,17 +36,19 @@ The output of `git rev-parse HEAD`: `XXXXX`
 The command you used to start the project: `XXXXX`
 
 The output of `docker-compose logs netbox`:
-<!-- if your log is very long, create a Gist instead: https://gist.github.com -->
-
-```
-LOG LOG LOG
-```
-
 <!--
-If you have get any 5xx http error, else delete this section.
-If your log is very long, create a Gist instead: https://gist.github.com
+If your log is very long, create a Gist instead (and post the link to it): https://gist.github.com
+-->
+
+```
+LOG LOG LOG
+```
+
+The output of `docker-compose logs nginx`:
+<!--
+Only if you have gotten a 5xx http error, else delete this section.
+If your log is very long, create a Gist instead (and post the link to it): https://gist.github.com
 -->
-The output of `docker-compose logs nginx`:
 
 ```
 LOG LOG LOG

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.6-alpine3.7
+FROM python:3.6-alpine3.8
 
 RUN apk add --no-cache \
       bash \
@@ -6,13 +6,13 @@ RUN apk add --no-cache \
       ca-certificates \
       cyrus-sasl-dev \
       graphviz \
-      ttf-ubuntu-font-family \
       jpeg-dev \
       libffi-dev \
       libxml2-dev \
       libxslt-dev \
       openldap-dev \
       postgresql-dev \
+      ttf-ubuntu-font-family \
       wget
 
 RUN pip install \
@@ -21,7 +21,13 @@ RUN pip install \
 # napalm is used for gathering information from network devices
       napalm \
 # ruamel is used in startup_scripts
-      ruamel.yaml
+      ruamel.yaml \
+# pinning django to the version required by netbox
+# adding it here, to install the correct version of
+# django-rq
+      'Django>=1.11,<2.1' \
+# django-rq is used for webhooks
+      django-rq
 
 WORKDIR /opt
 
@@ -45,8 +51,6 @@ WORKDIR /opt/netbox/netbox
 
 ENTRYPOINT [ "/opt/netbox/docker-entrypoint.sh" ]
 
-VOLUME ["/etc/netbox-nginx/"]
-
 CMD ["gunicorn", "-c /etc/netbox/config/gunicorn_config.py", "netbox.wsgi"]
 
 LABEL SRC_URL="$URL"

README.md

@@ -5,8 +5,11 @@
 This repository houses the components needed to build NetBox as a Docker container.
 Images built using this code are released to [Docker Hub][netbox-dockerhub] every night.
 
+Questions? Before opening an issue on Github, please join the [Network To Code][ntc-slack] and ask for help in our `#netbox-docker` channel.
+
 [travis]: https://travis-ci.org/ninech/netbox-docker
 [netbox-dockerhub]: https://hub.docker.com/r/ninech/netbox/tags/
+[ntc-slack]: http://slack.networktocode.com/
 
 ## Quickstart
 
@@ -54,7 +57,8 @@ To ensure this, compare the output of `docker --version` and `docker-compose --v
 
 ## Configuration
 
-You can configure the app using environment variables. These are defined in `netbox.env`.
+You can configure the app using environment variables.
+These are defined in `netbox.env`.
 Read [Environment Variables in Compose][compose-env] to understand about the various possibilities to overwrite these variables.
 (The easiest solution being simply adjusting that file.)
 
@@ -70,11 +74,12 @@ For example defining `ALLOWED_HOSTS=localhost ::1 127.0.0.1` would allows access
 The default settings are optimized for (local) development environments.
 You should therefore adjust the configuration for production setups, at least the following variables:
 
-* `ALLOWED_HOSTS`: Add all URLs that lead to your NetBox instance.
-* `DB_*`: Use a persistent database.
+* `ALLOWED_HOSTS`: Add all URLs that lead to your NetBox instance, space separated. E.g. `ALLOWED_HOSTS=netbox.mycorp.com server042.mycorp.com 2a02:123::42 10.0.0.42 localhost ::1 127.0.0.1` (It's good advice to always allow localhost connections for easy debugging, i.e. `localhost ::1 127.0.0.1`.)
+* `DB_*`: Use your own persistent database. Don't use the default passwords!
 * `EMAIL_*`: Use your own mailserver.
 * `MAX_PAGE_SIZE`: Use the recommended default of 1000.
-* `SUPERUSER_*`: Only define those variables during the initial setup, and drop them once the DB is set up.
+* `SUPERUSER_*`: Only define those variables during the initial setup, and drop them once the DB is set up. Don't use the default passwords!
+* `REDIS_*`: Use your own persistent redis. Don't use the default passwords!
 
 ### Running on Docker Swarm / Kubernetes / OpenShift
 
@@ -95,6 +100,7 @@ If a secret is defined by an environment variable and in the respective file at
 * `SECRET_KEY`: `/run/secrets/secret_key`
 * `EMAIL_PASSWORD`: `/run/secrets/email_password`
 * `NAPALM_PASSWORD`: `/run/secrets/napalm_password`
+* `REDIS_PASSWORD`: `/run/secrets/redis_password`
 
 Please also consider [the advice about running NetBox in production](#production) above!
 
@@ -125,6 +131,18 @@ However, if you don't need this functionality, leave these blank.
 [napalm-doc]: http://napalm.readthedocs.io/en/latest/index.html
 [netbox-napalm-doc]: https://netbox.readthedocs.io/en/latest/configuration/optional-settings/#napalm_username
 
+### Customizable Reporting
+
+NetBox includes [customized reporting][netbox-reports-doc] that allows the user to write Python code and determine the validity of the data within NetBox.
+The `REPORTS_ROOT` variable is setup as a mapped directory within this Docker container to `/reports/` and includes the example directly from the documentation for `devices.py`.
+However, it has been renamed to `devices.py.example` which prevents NetBox from recognizing it as a valid report.
+This was done to avoid unnessary issues from being opened when the default does not work for someone's expectations.
+
+To re-enable this default report, simply rename `devices.py.example` to `devices.py` and browse within the WebUI to `/extras/reports/`.
+You can also dynamically add any other report to this same directory and NetBox will be able to see it without restarting the container.
+
+[netbox-reports-doc]: https://netbox.readthedocs.io/en/stable/additional-features/reports/
+
 ### Custom Initialization Code (e.g. Automatically Setting Up Custom Fields)
 
 When using `docker-compose`, all the python scripts present in `/opt/netbox/startup_scripts` will automatically be executed after the application boots in the context of `./manage.py`.
@@ -155,7 +173,7 @@ if created:
 
 #### Initializers
 
-Initializers are built-in startup scripts for defining NetBox custom fields, groups and users.
+Initializers are built-in startup scripts for defining NetBox custom fields, groups, users and many other resources.
 All you need to do is to mount you own `initializers` folder ([see `docker-compose.yml`][netbox-docker-compose]).
 Look at the [`initializers` folder][netbox-docker-initializers] to learn how the files must look like.
 
@@ -182,6 +200,15 @@ text_field:
 [netbox-docker-initializers]: https://github.com/ninech/netbox-docker/tree/master/initializers
 [netbox-docker-compose]: https://github.com/ninech/netbox-docker/blob/master/docker-compose.yml
 
+##### Available Groups for User/Group initializers
+
+To get an up-to-date list about all the available permissions, run the following command.
+
+```bash
+# Make sure the 'netbox' container is already running! If unsure, run `docker-compose up -d`
+echo "from django.contrib.auth.models import Permission\nfor p in Permission.objects.all():\n  print(p.codename);" | docker-compose exec -T netbox ./manage.py shell
+```
+
 #### Custom Docker Image
 
 You can also build your own NetBox Docker image containing your own startup scripts, custom fields, users and groups
@@ -249,6 +276,32 @@ If your issue is not here, look through [the existing issues][issues] and eventu
 * To create a database backup run `docker-compose exec postgres sh -c 'pg_dump -cU $POSTGRES_USER $POSTGRES_DB' | gzip > db_dump.sql.gz`
 * To restore that database backup run `gunzip -c db_dump.sql.gz | docker exec -i $(docker-compose ps -q postgres) sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'`.
 
+### Nginx doesn't start
+
+As a first step, stop your docker-compose setup.
+Then locate the `netbox-nginx-config` volume and remove it:
+
+```bash
+# Stop your local netbox-docker installation
+$ docker-compose down
+
+# Find the volume
+$ docker volume ls | grep netbox-nginx-config
+local               netbox-docker_netbox-nginx-config
+
+# Remove the volume
+$ docker volume rm netbox-docker_netbox-nginx-config
+netbox-docker_netbox-nginx-config
+```
+
+Now start everything up again.
+
+If this didn't help, try to see if there's anything in the logs indicating why nginx doesn't start:
+
+```bash
+$ docker-compose logs -f nginx
+```
+
 ### Getting a "Bad Request (400)"
 
 > When connecting to the NetBox instance, I get a "Bad Request (400)" error.
@@ -257,26 +310,81 @@ This usually happens when the `ALLOWED_HOSTS` variable is not set correctly.
 
 ### How to upgrade
 
-> How do I update to a newer version?
+> How do I update to a newer version of netbox?
 
 It should be sufficient to pull the latest image from Docker Hub, stopping the container and starting it up again:
 
 ```bash
 docker-compose pull netbox
-docker-compose stop netbox
-docker-compose rm -f netbox
-docker-compose up -d netbox
+docker-compose stop netbox netbox-worker
+docker-compose rm -f netbox netbox-worker
+docker-compose up -d netbox netbox-worker
 ```
 
+### Webhooks don't work
+
+First make sure that the webhooks feature is enabled in your Netbox configuration and that a redis host is defined.
+Check `netbox.env` if the following variables are defined:
+
+```
+WEBHOOKS_ENABLED=true
+REDIS_HOST=redis
+```
+
+Then make sure that the `redis` container and at least one `netbox-worker` are running.
+
+```
+# check the container status
+$ docker-compose ps
+
+Name                           Command               State                Ports
+--------------------------------------------------------------------------------------------------------
+netbox-docker_netbox-worker_1   /opt/netbox/docker-entrypo ...   Up
+netbox-docker_netbox_1          /opt/netbox/docker-entrypo ...   Up
+netbox-docker_nginx_1           nginx -c /etc/netbox-nginx ...   Up      80/tcp, 0.0.0.0:32776->8080/tcp
+netbox-docker_postgres_1        docker-entrypoint.sh postgres    Up      5432/tcp
+netbox-docker_redis_1           docker-entrypoint.sh redis ...   Up      6379/tcp
+
+# connect to redis and send PING command:
+$ docker-compose run --rm -T redis sh -c 'redis-cli -h redis -a $REDIS_PASSWORD ping'
+Warning: Using a password with '-a' option on the command line interface may not be safe.
+PONG
+```
+
+If `redis` and the `netbox-worker` are not available, make sure you have updated your `docker-compose.yml` file!
+
+Everything's up and running? Then check the log of `netbox-worker` and/or `redis`:
+
+```bash
+docker-compose logs -f netbox-worker
+docker-compose logs -f redis
+```
+
+Still no clue? You can connect to the `redis` container and have it report any command that is currently executed on the server:
+
+```bash
+docker-compose run --rm -T redis sh -c 'redis-cli -h redis -a $REDIS_PASSWORD monitor'
+
+# Hit CTRL-C a few times to leave
+```
+
+If you don't see anything happening after you triggered a webhook, double-check the configuration of the `netbox` and the `netbox-worker` containers and also check the configuration of your webhook in the admin interface of Netbox.
+
 ### Breaking Changes
 
-From time to time it might become necessary to re-order the structure of the container.
-Things like the `docker-compose.yml` file or your Kubernets or OpenShift configurations have to be adjusted as a consequence.
+From time to time it might become necessary to re-engineer the structure of this setup.
+Things like the `docker-compose.yml` file or your Kubernetes or OpenShift configurations have to be adjusted as a consequence.
 Since April 2018 each image built from this repo contains a `NETBOX_DOCKER_PROJECT_VERSION` label.
 You can check the label of your local image by running `docker inspect ninech/netbox:v2.3.1 --format "{{json .ContainerConfig.Labels}}"`.
+Compare the version with the list below to check whether a breaking change was introduced with that version.
 
-The following is a list of breaking changes:
+The following is a list of breaking changes of the `netbox-docker` project:
 
+* 0.7.0: The value of the `MAX_PAGE_SIZE` environment variable was changed to `1000`, which is the default of Netbox.
+* 0.6.0: The naming of the default startup_scripts were changed.
+  If you overwrite them, you may need to adjust these scripts.
+* 0.5.0: Alpine was updated to 3.8, `*.env` moved to `/env` folder
+* 0.4.0: In order to use Netbox webhooks you need to add Redis and a netbox-worker to your docker-compose.yml.
 * 0.3.0: Field `filterable: <boolean` was replaced with field `filter_logic: loose/exact/disabled`. It will default to `CF_FILTER_LOOSE=loose` when not defined.
 * 0.2.0: Re-organized paths: `/etc/netbox -> /etc/netbox/config` and `/etc/reports -> /etc/netbox/reports`. Fixes [#54](https://github.com/ninech/netbox-docker/issues/54).
 * 0.1.0: Introduction of the `NETBOX_DOCKER_PROJECT_VERSION`. (Not a breaking change per se.)
@@ -304,9 +412,15 @@ You can use the following ENV variables to customize the build:
            Default: https://github.com/${SRC_REPO}/netbox/archive/$BRANCH.tar.gz
 ```
 
+### Publishing Docker Images
+
+New Docker Images are built and published every 24h by using travis:
+
+[![Build Status](https://travis-ci.org/ninech/netbox-docker.svg?branch=master)][travis]
+
 ## Tests
 
-To run the test coming with NetBox, use the `docker-compose.yml` file as such:
+To run the tests coming with NetBox, use the `docker-compose.yml` file as such:
 
 ```
 $ docker-compose run netbox ./manage.py test

VERSION

@@ -1 +1 @@
-0.3.1
+0.7.0

build-branches.sh

@@ -3,9 +3,17 @@
 
 echo "▶️ $0 $*"
 
+if [ ! -z "${GITHUB_OAUTH_CLIENT_ID}" ] && [ ! -z "${GITHUB_OAUTH_CLIENT_SECRET}" ]; then
+  echo "🗝 Performing authenticated Github API calls."
+  GITHUB_OAUTH_PARAMS="client_id=${GITHUB_OAUTH_CLIENT_ID}&client_secret=${GITHUB_OAUTH_CLIENT_SECRET}"
+else
+  echo "🕶 Performing unauthenticated Github API calls. This might result in lower Github rate limits!"
+  GITHUB_OAUTH_PARAMS=""
+fi
+
 ORIGINAL_GITHUB_REPO="digitalocean/netbox"
 GITHUB_REPO="${GITHUB_REPO-$ORIGINAL_GITHUB_REPO}"
-URL_RELEASES="https://api.github.com/repos/${GITHUB_REPO}/branches"
+URL_RELEASES="https://api.github.com/repos/${GITHUB_REPO}/branches?${GITHUB_OAUTH_PARAMS}"
 
 CURL="curl -sS"
 

build-latest.sh

@@ -3,9 +3,17 @@
 
 echo "▶️ $0 $*"
 
+if [ ! -z "${GITHUB_OAUTH_CLIENT_ID}" ] && [ ! -z "${GITHUB_OAUTH_CLIENT_SECRET}" ]; then
+  echo "🗝 Performing authenticated Github API calls."
+  GITHUB_OAUTH_PARAMS="client_id=${GITHUB_OAUTH_CLIENT_ID}&client_secret=${GITHUB_OAUTH_CLIENT_SECRET}"
+else
+  echo "🕶 Performing unauthenticated Github API calls. This might result in lower Github rate limits!"
+  GITHUB_OAUTH_PARAMS=""
+fi
+
 ORIGINAL_GITHUB_REPO="digitalocean/netbox"
 GITHUB_REPO="${GITHUB_REPO-$ORIGINAL_GITHUB_REPO}"
-URL_RELEASES="https://api.github.com/repos/${GITHUB_REPO}/releases"
+URL_RELEASES="https://api.github.com/repos/${GITHUB_REPO}/releases?${GITHUB_OAUTH_PARAMS}"
 
 JQ_LATEST="group_by(.prerelease) | .[] | sort_by(.published_at) | reverse | .[0] | select(.prerelease==${PRERELEASE-false}) | .tag_name"
 

configuration/configuration.py

@@ -139,6 +139,19 @@ PAGINATE_COUNT = int(os.environ.get('PAGINATE_COUNT', 50))
 # prefer IPv4 instead.
 PREFER_IPV4 = os.environ.get('PREFER_IPV4', 'False').lower() == 'true'
 
+# The Webhook event backend is disabled by default. Set this to True to enable it. Note that this requires a Redis
+# database be configured and accessible by NetBox (see `REDIS` below).
+WEBHOOKS_ENABLED = os.environ.get('WEBHOOKS_ENABLED', 'False').lower() == 'true'
+
+# Redis database settings (optional). A Redis database is required only if the webhooks backend is enabled.
+REDIS = {
+    'HOST': os.environ.get('REDIS_HOST', 'localhost'),
+    'PORT': os.environ.get('REDIS_PORT', 6379),
+    'PASSWORD': os.environ.get('REDIS_PASSWORD', read_secret('redis_password')),
+    'DATABASE': os.environ.get('REDIS_DATABASE', '0'),
+    'DEFAULT_TIMEOUT': os.environ.get('REDIS_TIMEOUT', '300'),
+}
+
 # The file path where custom reports will be stored. A trailing slash is not needed. Note that the default value of
 # this setting is derived from the installed location.
 REPORTS_ROOT = os.environ.get('REPORTS_ROOT', '/etc/netbox/reports')

docker-compose.yml

@@ -1,46 +1,67 @@
 version: '3'
 services:
-    netbox:
-        build:
-          context: .
-          args:
-            - BRANCH=${VERSION-master}
-        image: ninech/netbox:${VERSION-latest}
-        depends_on:
-        - postgres
-        env_file: netbox.env
-        volumes:
-        - ./startup_scripts:/opt/netbox/startup_scripts:ro
-        - ./initializers:/opt/netbox/initializers:ro
-        - ./configuration:/etc/netbox/config:ro
-        - netbox-nginx-config:/etc/netbox-nginx/
-        - netbox-static-files:/opt/netbox/netbox/static
-        - netbox-media-files:/opt/netbox/netbox/media
-        - netbox-report-files:/etc/netbox/reports:ro
-    nginx:
-        image: nginx:1.13-alpine
-        command: nginx -g 'daemon off;' -c /etc/netbox-nginx/nginx.conf
-        depends_on:
-        - netbox
-        ports:
-        - 8080
-        volumes:
-        - netbox-static-files:/opt/netbox/netbox/static:ro
-        - netbox-nginx-config:/etc/netbox-nginx/:ro
-    postgres:
-        image: postgres:10.4-alpine
-        env_file: postgres.env
-        volumes:
-        - netbox-postgres-data:/var/lib/postgresql/data
-
+  netbox: &netbox
+    build:
+      context: .
+      args:
+        - BRANCH=${VERSION-master}
+    image: ninech/netbox:${VERSION-latest}
+    depends_on:
+    - postgres
+    - redis
+    - netbox-worker
+    env_file: env/netbox.env
+    volumes:
+    - ./startup_scripts:/opt/netbox/startup_scripts:ro
+    - ./initializers:/opt/netbox/initializers:ro
+    - ./configuration:/etc/netbox/config:ro
+    - ./reports:/etc/netbox/reports:ro
+    - netbox-nginx-config:/etc/netbox-nginx/
+    - netbox-static-files:/opt/netbox/netbox/static
+    - netbox-media-files:/opt/netbox/netbox/media
+  netbox-worker:
+    <<: *netbox
+    depends_on:
+    - redis
+    entrypoint:
+    - python3
+    - /opt/netbox/netbox/manage.py
+    command:
+    - rqworker
+  nginx:
+    command: nginx -c /etc/netbox-nginx/nginx.conf
+    image: nginx:1.15-alpine
+    depends_on:
+    - netbox
+    ports:
+    - 8080
+    volumes:
+    - netbox-static-files:/opt/netbox/netbox/static:ro
+    - netbox-nginx-config:/etc/netbox-nginx/:ro
+  postgres:
+    image: postgres:10.4-alpine
+    env_file: env/postgres.env
+    volumes:
+    - netbox-postgres-data:/var/lib/postgresql/data
+  redis:
+    image: redis:4-alpine
+    command:
+    - sh
+    - -c # this is to evaluate the $REDIS_PASSWORD from the env
+    - redis-server --appendonly yes --requirepass $$REDIS_PASSWORD ## $$ because of docker-compose
+    env_file: env/redis.env
+    volumes:
+    - netbox-redis-data:/data
 volumes:
-    netbox-static-files:
-        driver: local
-    netbox-nginx-config:
-        driver: local
-    netbox-media-files:
-        driver: local
-    netbox-report-files:
-        driver: local
-    netbox-postgres-data:
-        driver: local
+  netbox-static-files:
+    driver: local
+  netbox-nginx-config:
+    driver: local
+  netbox-media-files:
+    driver: local
+  netbox-report-files:
+    driver: local
+  netbox-postgres-data:
+    driver: local
+  netbox-redis-data:
+    driver: local

docker/nginx.conf

@@ -1,5 +1,8 @@
+daemon off;
 worker_processes 1;
 
+error_log /dev/stderr info;
+
 events {
     worker_connections 1024;
 }
@@ -16,7 +19,6 @@ http {
 
     server {
         listen      8080;
-        server_name localhost;
         access_log  off;
 
         location /static/ {

env/netbox.env

@@ -12,9 +12,12 @@ MEDIA_ROOT=/opt/netbox/netbox/media
 NAPALM_USERNAME=
 NAPALM_PASSWORD=
 NAPALM_TIMEOUT=10
-MAX_PAGE_SIZE=0
+MAX_PAGE_SIZE=1000
+REDIS_HOST=redis
+REDIS_PASSWORD=H733Kdjndks81
 SECRET_KEY=r8OwDznj!!dci#P9ghmRfdu1Ysxm0AiPeDCQhKE+N_rClfWNj
 SUPERUSER_NAME=admin
 SUPERUSER_EMAIL=admin@example.com
 SUPERUSER_PASSWORD=admin
 SUPERUSER_API_TOKEN=0123456789abcdef0123456789abcdef01234567
+WEBHOOKS_ENABLED=true

env/redis.env

@@ -0,0 +1 @@
+REDIS_PASSWORD=H733Kdjndks81

initializers/custom_fields.yml

@@ -7,6 +7,8 @@
 #   on_objects:
 #   - dcim.models.Device
 #   - dcim.models.Rack
+#   - dcim.models.Site
+#   - dcim.models.DeviceType
 #   - ipam.models.IPAddress
 #   - ipam.models.Prefix
 #   - tenancy.models.Tenant

initializers/device_roles.yml

@@ -0,0 +1,15 @@
+# - name: switch
+#   slug: switch
+#   color: Grey
+# - name: router
+#   slug: router
+#   color: Cyan
+# - name: load-balancer
+#   slug: load-balancer
+#   color: Red
+# - name: server
+#   slug: server
+#   color: Blue
+# - name: patchpanel
+#   slug: patchpanel
+#   color: Black

initializers/device_types.yml

@@ -0,0 +1,23 @@
+# - model: Model 1
+#   manufacturer: Manufacturer 1
+#   slug: model-1
+#   u_height: 2
+#   custom_fields:
+#     text_field: Description
+# - model: Model 2
+#   manufacturer: Manufacturer 1
+#   slug: model-2
+#   custom_fields:
+#     text_field: Description
+# - model: Model 3
+#   manufacturer: Manufacturer 1
+#   slug: model-3
+#   is_full_depth: false
+#   u_height: 0
+#   custom_fields:
+#     text_field: Description
+# - model: Other
+#   manufacturer: NoName
+#   slug: other
+#   custom_fields:
+#     text_field: Description

initializers/devices.yml

@@ -0,0 +1,27 @@
+# - name: server01
+#   device_role: server
+#   device_type: Other
+#   site: AMS 1
+#   rack: rack-01
+#   face: Front
+#   position: 1
+#   custom_fields:
+#     text_field: Description
+# - name: server02
+#   device_role: server
+#   device_type: Other
+#   site: AMS 2
+#   rack: rack-02
+#   face: Front
+#   position: 2
+#   custom_fields:
+#     text_field: Description
+# - name: server03
+#   device_role: server
+#   device_type: Other
+#   site: SING 1
+#   rack: rack-03
+#   face: Front
+#   position: 3
+#   custom_fields:
+#     text_field: Description

initializers/groups.yml

@@ -7,3 +7,10 @@
 # writers:
 #   users:
 #   - writer
+#   permissions:
+#   - add_device
+#   - change_device
+#   - delete_device
+#   - add_virtualmachine
+#   - change_virtualmachine
+#   - delete_virtualmachine

initializers/manufacturers.yml

@@ -0,0 +1,6 @@
+# - name: Manufacturer 1
+#   slug: manufacturer-1
+# - name: Manufacturer 2
+#   slug: manufacturer-2
+# - name: NoName
+#   slug: noname

initializers/platforms.yml

@@ -0,0 +1,19 @@
+# # Allowed rpc clients are: juniper-junos, cisco-ios, opengear
+# - name: Platform 1
+#   slug: platform-1
+#   manufacturer: Manufacturer 1
+#   napalm_driver: driver1
+#   napalm_args: "{'arg1': 'value1', 'arg2': 'value2'}"
+#   rpc_client: juniper-junos
+# - name: Platform 2
+#   slug: platform-2
+#   manufacturer: Manufacturer 2
+#   napalm_driver: driver2
+#   napalm_args: "{'arg1': 'value1', 'arg2': 'value2'}"
+#   rpc_client: opengear
+# - name: Platform 3
+#   slug: platform-3
+#   manufacturer: NoName
+#   napalm_driver: driver3
+#   napalm_args: "{'arg1': 'value1', 'arg2': 'value2'}"
+#   rpc_client: juniper-junos

initializers/rack_roles.yml

@@ -0,0 +1,12 @@
+# - name: Role 1
+#   slug: role-1
+#   color: Pink
+# - name: Role 2
+#   slug: role-2
+#   color: Cyan
+# - name: Role 3
+#   slug: role-3
+#   color: Grey
+# - name: Role 4
+#   slug: role-4
+#   color: Teal

initializers/racks.yml

@@ -0,0 +1,24 @@
+# - site: AMS 1
+#   name: rack-01
+#   role: Role 1
+#   type: 4-post cabinet
+#   width: 19 inches
+#   u_height: 47
+#   custom_fields:
+#     text_field: Description
+# - site: AMS 2
+#   name: rack-02
+#   role: Role 2
+#   type: 4-post cabinet
+#   width: 19 inches
+#   u_height: 47
+#   custom_fields:
+#     text_field: Description
+# - site: SING 1
+#   name: rack-03
+#   role: Role 3
+#   type: 4-post cabinet
+#   width: 19 inches
+#   u_height: 47
+#   custom_fields:
+#     text_field: Description

initializers/regions.yml

@@ -0,0 +1,10 @@
+# - name: Singapore
+#   slug: singapore
+# - name: Amsterdam
+#   slug: amsterdam
+# - name: Downtown
+#   slug: downtown
+#   parent: Amsterdam
+# - name: Suburbs
+#   slug: suburbs
+#   parent: Amsterdam

initializers/sites.yml

@@ -0,0 +1,32 @@
+# - name: AMS 1
+#   slug: ams1
+#   region: Downtown
+#   status: 1
+#   facility: Amsterdam 1
+#   asn: 12345
+#   custom_fields:
+#     text_field: Description
+# - name: AMS 2
+#   slug: ams2
+#   region: Downtown
+#   status: 1
+#   facility: Amsterdam 2
+#   asn: 54321
+#   custom_fields:
+#     text_field: Description
+# - name: AMS 3
+#   slug: ams3
+#   region: Suburbs
+#   status: 1
+#   facility: Amsterdam 3
+#   asn: 67890
+#   custom_fields:
+#     text_field: Description
+# - name: SING 1
+#   slug: sing1
+#   region: Singapore
+#   status: 1
+#   facility: Singapore 1
+#   asn: 09876
+#   custom_fields:
+#     text_field: Description

initializers/users.yml

@@ -4,3 +4,10 @@
 #   password: reader
 # writer:
 #   password: writer
+#   permissions:
+#   - add_device
+#   - change_device
+#   - delete_device
+#   - add_virtualmachine
+#   - change_virtualmachine
+#   - delete_virtualmachine

reports/devices.py.example

@@ -0,0 +1,46 @@
+from dcim.constants import CONNECTION_STATUS_PLANNED, DEVICE_STATUS_ACTIVE
+from dcim.models import ConsolePort, Device, PowerPort
+from extras.reports import Report
+
+
+class DeviceConnectionsReport(Report):
+    description = "Validate the minimum physical connections for each device"
+
+    def test_console_connection(self):
+
+        # Check that every console port for every active device has a connection defined.
+        for console_port in ConsolePort.objects.select_related('device').filter(device__status=DEVICE_STATUS_ACTIVE):
+            if console_port.connected_endpoint is None:
+                self.log_failure(
+                    console_port.device,
+                    "No console connection defined for {}".format(console_port.name)
+                )
+            elif console_port.connection_status == CONNECTION_STATUS_PLANNED:
+                self.log_warning(
+                    console_port.device,
+                    "Console connection for {} marked as planned".format(console_port.name)
+                )
+            else:
+                self.log_success(console_port.device)
+
+    def test_power_connections(self):
+
+        # Check that every active device has at least two connected power supplies.
+        for device in Device.objects.filter(status=DEVICE_STATUS_ACTIVE):
+            connected_ports = 0
+            for power_port in PowerPort.objects.filter(device=device):
+                if power_port.connected_endpoint is not None:
+                    connected_ports += 1
+                    if power_port.connection_status == CONNECTION_STATUS_PLANNED:
+                        self.log_warning(
+                            device,
+                            "Power connection for {} marked as planned".format(power_port.name)
+                        )
+            if connected_ports < 2:
+                self.log_failure(
+                    device,
+                    "{} connected power supplies found (2 needed)".format(connected_ports)
+                )
+            else:
+                self.log_success(device)
+

startup_scripts/000_users.py

@@ -0,0 +1,34 @@
+from django.contrib.auth.models import Permission, Group, User
+from users.models import Token
+
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/users.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml=YAML(typ='safe')
+  users = yaml.load(stream)
+
+  if users is not None:
+    for username, user_details in users.items():
+      if not User.objects.filter(username=username):
+        user = User.objects.create_user(
+          username = username,
+          password = user_details.get('password', 0) or User.objects.make_random_password)
+
+        print("👤 Created user ",username)
+
+        if user_details.get('api_token', 0):
+          Token.objects.create(user=user, key=user_details['api_token'])
+
+        user_permissions = user_details.get('permissions', [])
+        if user_permissions:
+          user.user_permissions.clear()
+          for permission_codename in user_details.get('permissions', []):
+            permission = Permission.objects.get(codename=permission_codename)
+            user.user_permissions.add(permission)
+          user.save()

startup_scripts/00_users.py

@@ -1,20 +0,0 @@
-from django.contrib.auth.models import Group, User
-from users.models import Token
-
-from ruamel.yaml import YAML
-
-with open('/opt/netbox/initializers/users.yml', 'r') as stream:
-  yaml=YAML(typ='safe')
-  users = yaml.load(stream)
-
-  if users is not None:
-    for username, user_details in users.items():
-      if not User.objects.filter(username=username):
-        user = User.objects.create_user(
-          username = username,
-          password = user_details.get('password', 0) or User.objects.make_random_password)
-
-        print("👤 Created user ",username)
-
-        if user_details.get('api_token', 0):
-          Token.objects.create(user=user, key=user_details['api_token'])

startup_scripts/010_groups.py

@@ -0,0 +1,33 @@
+from django.contrib.auth.models import Permission, Group, User
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/groups.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml=YAML(typ='safe')
+  groups = yaml.load(stream)
+
+  if groups is not None:
+    for groupname, group_details in groups.items():
+      group, created = Group.objects.get_or_create(name=groupname)
+
+      if created:
+        print("👥 Created group", groupname)
+
+      for username in group_details.get('users', []):
+        user = User.objects.get(username=username)
+
+        if user:
+          user.groups.add(group)
+
+      group_permissions = group_details.get('permissions', [])
+      if group_permissions:
+        group.permissions.clear()
+        print("Permissions:", group.permissions.all())
+        for permission_codename in group_details.get('permissions', []):
+          permission = Permission.objects.get(codename=permission_codename)
+          group.permissions.add(permission)

startup_scripts/020_custom_fields.py

@@ -2,6 +2,8 @@ from extras.constants import CF_TYPE_TEXT, CF_TYPE_INTEGER, CF_TYPE_BOOLEAN, CF_
 from extras.models import CustomField, CustomFieldChoice
 
 from ruamel.yaml import YAML
+from pathlib import Path
+import sys
 
 text_to_fields = {
   'boolean': CF_TYPE_BOOLEAN,
@@ -21,7 +23,11 @@ def get_class_for_class_path(class_path):
   clazz = getattr(module, class_name)
   return ContentType.objects.get_for_model(clazz)
 
-with open('/opt/netbox/initializers/custom_fields.yml', 'r') as stream:
+file = Path('/opt/netbox/initializers/custom_fields.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
   yaml = YAML(typ='safe')
   customfields = yaml.load(stream)
 

startup_scripts/030_regions.py

@@ -0,0 +1,31 @@
+from dcim.models import Region
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/regions.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml=YAML(typ='safe')
+  regions = yaml.load(stream)
+
+  optional_assocs = {
+    'parent': (Region, 'name')
+  }
+
+  if regions is not None:
+    for params in regions:
+
+      for assoc, details in optional_assocs.items():
+        if assoc in params:
+          model, field = details
+          query = { field: params.pop(assoc) }
+
+          params[assoc] = model.objects.get(**query)
+
+      region, created = Region.objects.get_or_create(**params)
+
+      if created:
+        print("🌐 Created region", region.name)

startup_scripts/040_sites.py

@@ -0,0 +1,46 @@
+from dcim.models import Region, Site
+from extras.models import CustomField, CustomFieldValue
+from tenancy.models import Tenant
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/sites.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml = YAML(typ='safe')
+  sites = yaml.load(stream)
+
+  optional_assocs = {
+    'region': (Region, 'name'),
+    'tenant': (Tenant, 'name')
+  }
+
+  if sites is not None:
+    for params in sites:
+      custom_fields = params.pop('custom_fields', None)
+
+      for assoc, details in optional_assocs.items():
+        if assoc in params:
+          model, field = details
+          query = { field: params.pop(assoc) }
+
+          params[assoc] = model.objects.get(**query)
+
+      site, created = Site.objects.get_or_create(**params)
+
+      if created:
+        if custom_fields is not None:
+          for cf_name, cf_value in custom_fields.items():
+            custom_field = CustomField.objects.get(name=cf_name)
+            custom_field_value = CustomFieldValue.objects.create(
+              field=custom_field,
+              obj=site,
+              value=cf_value
+            )
+
+            site.custom_field_values.add(custom_field_value)
+
+        print("📍 Created site", site.name)

startup_scripts/050_manufacturers.py

@@ -0,0 +1,19 @@
+from dcim.models import Manufacturer
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/manufacturers.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml = YAML(typ='safe')
+  manufacturers = yaml.load(stream)
+
+  if manufacturers is not None:
+    for params in manufacturers:
+      manufacturer, created = Manufacturer.objects.get_or_create(**params)
+
+      if created:
+        print("🏭 Created Manufacturer", manufacturer.name)

startup_scripts/060_device_types.py

@@ -0,0 +1,56 @@
+from dcim.models import DeviceType, Manufacturer, Region
+from tenancy.models import Tenant
+from extras.models import CustomField, CustomFieldValue
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/device_types.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml = YAML(typ='safe')
+  device_types = yaml.load(stream)
+
+  required_assocs = {
+    'manufacturer': (Manufacturer, 'name')
+  }
+
+  optional_assocs = {
+    'region': (Region, 'name'),
+    'tenant': (Tenant, 'name')
+  }
+
+  if device_types is not None:
+    for params in device_types:
+      custom_fields = params.pop('custom_fields', None)
+
+      for assoc, details in required_assocs.items():
+        model, field = details
+        query = { field: params.pop(assoc) }
+
+        params[assoc] = model.objects.get(**query)
+
+      for assoc, details in optional_assocs.items():
+        if assoc in params:
+          model, field = details
+          query = { field: params.pop(assoc) }
+
+          params[assoc] = model.objects.get(**query)
+
+      device_type, created = DeviceType.objects.get_or_create(**params)
+
+      if created:
+        if custom_fields is not None:
+          for cf_name, cf_value in custom_fields.items():
+            custom_field = CustomField.objects.get(name=cf_name)
+            custom_field_value = CustomFieldValue.objects.create(
+              field=custom_field,
+              obj=device_type,
+              value=cf_value
+            )
+
+            device_type.custom_field_values.add(custom_field_value)
+
+        print("🔡 Created device type", device_type.manufacturer, device_type.model)

startup_scripts/070_rack_roles.py

@@ -0,0 +1,28 @@
+from dcim.models import RackRole
+from ruamel.yaml import YAML
+from utilities.forms import COLOR_CHOICES
+
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/rack_roles.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml=YAML(typ='safe')
+  rack_roles = yaml.load(stream)
+
+  if rack_roles is not None:
+    for params in rack_roles:
+      if 'color' in params:
+        color = params.pop('color')
+
+        for color_tpl in COLOR_CHOICES:
+          if color in color_tpl:
+            params['color'] = color_tpl[0]
+
+      rack_role, created = RackRole.objects.get_or_create(**params)
+
+      if created:
+        print("🎨 Created rack role", rack_role.name)

startup_scripts/080_racks.py

@@ -0,0 +1,66 @@
+from dcim.models import Site, RackRole, Rack, RackGroup
+from tenancy.models import Tenant
+from extras.models import CustomField, CustomFieldValue
+from dcim.constants import RACK_TYPE_CHOICES, RACK_WIDTH_CHOICES
+from ruamel.yaml import YAML
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/racks.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml = YAML(typ='safe')
+  racks = yaml.load(stream)
+
+  required_assocs = {
+    'site': (Site, 'name')
+  }
+
+  optional_assocs = {
+    'role': (RackRole, 'name'),
+    'tenant': (Tenant, 'name'),
+    'group': (RackGroup, 'name')
+  }
+
+  if racks is not None:
+    for params in racks:
+      custom_fields = params.pop('custom_fields', None)
+
+      for assoc, details in required_assocs.items():
+        model, field = details
+        query = { field: params.pop(assoc) }
+
+        params[assoc] = model.objects.get(**query)
+
+      for assoc, details in optional_assocs.items():
+        if assoc in params:
+          model, field = details
+          query = { field: params.pop(assoc) }
+
+          params[assoc] = model.objects.get(**query)
+
+      for rack_type in RACK_TYPE_CHOICES:
+        if params['type'] in rack_type:
+          params['type'] = rack_type[0]
+
+      for rack_width in RACK_WIDTH_CHOICES:
+        if params['width'] in rack_width:
+          params['width'] = rack_width[0]
+
+      rack, created = Rack.objects.get_or_create(**params)
+
+      if created:
+        if custom_fields is not None:
+          for cf_name, cf_value in custom_fields.items():
+            custom_field = CustomField.objects.get(name=cf_name)
+            custom_field_value = CustomFieldValue.objects.create(
+              field=custom_field,
+              obj=rack,
+              value=cf_value
+            )
+
+            rack.custom_field_values.add(custom_field_value)
+
+        print("🔳 Created rack", rack.site, rack.name)

startup_scripts/090_device_roles.py

@@ -0,0 +1,29 @@
+from dcim.models import DeviceRole
+from ruamel.yaml import YAML
+from utilities.forms import COLOR_CHOICES
+
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/device_roles.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml=YAML(typ='safe')
+  device_roles = yaml.load(stream)
+
+  if device_roles is not None:
+    for params in device_roles:
+
+      if 'color' in params:
+        color = params.pop('color')
+
+        for color_tpl in COLOR_CHOICES:
+          if color in color_tpl:
+            params['color'] = color_tpl[0]
+
+      device_role, created = DeviceRole.objects.get_or_create(**params)
+
+      if created:
+        print("🎨 Created device role", device_role.name)

startup_scripts/100_platforms.py

@@ -0,0 +1,32 @@
+from dcim.models import Manufacturer, Platform
+from ruamel.yaml import YAML
+
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/platforms.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml = YAML(typ='safe')
+  platforms = yaml.load(stream)
+
+  optional_assocs = {
+    'manufacturer': (Manufacturer, 'name'),
+  }
+
+  if platforms is not None:
+    for params in platforms:
+
+      for assoc, details in optional_assocs.items():
+        if assoc in params:
+          model, field = details
+          query = { field: params.pop(assoc) }
+
+          params[assoc] = model.objects.get(**query)
+
+      platform, created = Platform.objects.get_or_create(**params)
+
+      if created:
+        print("💾 Created platform", platform.name)

startup_scripts/10_groups.py

@@ -1,19 +0,0 @@
-from django.contrib.auth.models import Group, User
-from ruamel.yaml import YAML
-
-with open('/opt/netbox/initializers/groups.yml', 'r') as stream:
-  yaml=YAML(typ='safe')
-  groups = yaml.load(stream)
-
-  if groups is not None:
-    for groupname, group_details in groups.items():
-      group, created = Group.objects.get_or_create(name=groupname)
-
-      if created:
-        print("👥 Created group", groupname)
-
-      for username in group_details['users']:
-        user = User.objects.get(username=username)
-
-        if user:
-          user.groups.add(group)

startup_scripts/110_devices.py

@@ -0,0 +1,71 @@
+from dcim.models import Site, Rack, DeviceRole, DeviceType, Device, Platform
+from dcim.constants import RACK_FACE_CHOICES
+from ipam.models import IPAddress
+from virtualization.models import Cluster
+from tenancy.models import Tenant
+from extras.models import CustomField, CustomFieldValue
+from ruamel.yaml import YAML
+
+from pathlib import Path
+import sys
+
+file = Path('/opt/netbox/initializers/devices.yml')
+if not file.is_file():
+  sys.exit()
+
+with file.open('r') as stream:
+  yaml = YAML(typ='safe')
+  devices = yaml.load(stream)
+
+  required_assocs = {
+    'device_role': (DeviceRole, 'name'),
+    'device_type': (DeviceType, 'model'),
+    'site': (Site, 'name')
+  }
+
+  optional_assocs = {
+    'tenant': (Tenant, 'name'),
+    'platform': (Platform, 'name'),
+    'rack': (Rack, 'name'),
+    'cluster': (Cluster, 'name'),
+    'primary_ip4': (IPAddress, 'address'),
+    'primary_ip6': (IPAddress, 'address')
+  }
+
+  if devices is not None:
+    for params in devices:
+      custom_fields = params.pop('custom_fields', None)
+
+      for assoc, details in required_assocs.items():
+        model, field = details
+        query = { field: params.pop(assoc) }
+
+        params[assoc] = model.objects.get(**query)
+
+      for assoc, details in optional_assocs.items():
+        if assoc in params:
+          model, field = details
+          query = { field: params.pop(assoc) }
+
+          params[assoc] = model.objects.get(**query)
+
+      if 'face' in params:
+        for rack_face in RACK_FACE_CHOICES:
+          if params['face'] in rack_face:
+            params['face'] = rack_face[0]
+
+      device, created = Device.objects.get_or_create(**params)
+
+      if created:
+        if custom_fields is not None:
+          for cf_name, cf_value in custom_fields.items():
+            custom_field = CustomField.objects.get(name=cf_name)
+            custom_field_value = CustomFieldValue.objects.create(
+              field=custom_field,
+              obj=device,
+              value=cf_value
+            )
+
+            device.custom_field_values.add(custom_field_value)
+
+        print("🖥️  Created device", device.name)