Merge pull request #239 from netbox-community/develop

Release 0.22.0

.github/ISSUE_TEMPLATE/bug_report.md

@@ -48,7 +48,7 @@ The output of `git rev-parse HEAD`: `XXXXX`
 The command you used to start the project: `XXXXX`
 
 <!-- adjust the `latest` tag to the version you're using -->
-The output of `docker inspect netboxcommunity/netbox:latest --format "{{json .ContainerConfig.Labels}}"`:
+The output of `docker inspect netboxcommunity/netbox:latest --format "{{json .Config.Labels}}"`:
 
 ```json
 {

.github/pull_request_template.md

@@ -1,5 +1,5 @@
 <!--
-###############################################################################
+#########################################################################
 
 Thank you for sharing your work and for opening a PR.
 
@@ -9,11 +9,11 @@ First make sure that you point your PR to the `develop` branch!
 Now please read the comments carefully and try to provide information
 on all relevant titles.
 
-###############################################################################
+#########################################################################
 -->
 
 <!--
-Please don't open an extra issue when submiting a PR.
+Please don't open an extra issue when submitting a PR.
 
 But if there is already a related issue, please put it's number here.
 
@@ -81,5 +81,5 @@ Please put an x into the brackets (like `[x]`) if you've completed that task.
 -->
 
 * [ ] I have read the comments and followed the PR template.
-* [ ] I have provided and explained my PR according to the information in the comments.
+* [ ] I have explained my PR according to the information in the comments.
 * [ ] My PR targets the `develop` branch.

.github/workflows/push.yml

@@ -1,7 +1,12 @@
+name: push
+
 on:
   push:
     branches-ignore:
       - release
+  pull_request:
+    branches-ignore:
+      - release
 
 jobs:
   build:
@@ -15,7 +20,7 @@ jobs:
         - '' # use the default of the DOCKERFILE
         - python:3.7-alpine
         - python:3.8-alpine
-        - python:3.9-rc-alpine
+        # - python:3.9-rc-alpine # disable until Netbox's unit tests work
       fail-fast: false
     runs-on: ubuntu-latest
     name: Builds new Netbox Docker Images

.github/workflows/release.yml

@@ -1,3 +1,5 @@
+name: release
+
 on:
   push:
     branches:

DOCKER_HUB.md

@@ -1,63 +0,0 @@
-# cloud.docker.com Configuration
-
-The automatic build is configured in cloud.docker.com.
-
-The following build configuration is expected:
-
-```yaml
-Source Repository: github.com/netbox-community/netbox-docker
-Build Location: Build on Docker Hub's infrastructure
-Autotest: Internal and External Pull Requests
-Repository Links: Enable for Base Image
-Build Rules:
-- Source Type: Branch
-  Source: release
-  Docker Tag: branches
-  Dockerfile location: Dockerfile
-  Build Context: /
-  Autobuild: on
-  Build Caching: on
-- Source Type: Branch
-  Source: release
-  Docker Tag: prerelease
-  Dockerfile location: Dockerfile
-  Build Context: /
-  Autobuild: on
-  Build Caching: on
-- Source Type: Branch
-  Source: release
-  Docker Tag: release
-  Dockerfile location: Dockerfile
-  Build Context: /
-  Autobuild: on
-  Build Caching: on
-Build Environment Variables:
-# Create an app on Github and use it's OATH credentials here
-- Key: GITHUB_OAUTH_CLIENT_ID
-  Value: <secret>
-- Key: GITHUB_OAUTH_CLIENT_SECRET
-  Value: <secret>
-Build Triggers:
-- Name: Cron Trigger
-  Trigger URL: <generated>
-# Use this trigger in combination with e.g. https://cron-job.org in order to regularly schedule builds
-```
-
-## Background Knowledge
-
-The build system of cloud.docker.com is not made for this kind of project.
-But we found a way to make it work, and this is how:
-
-1. The docker hub build system [allows to overwrite the scripts that get executed
-   for `build`, `test` and `push`](overwrite). See `/hooks/*`.
-2. Shared functionality of the scripts `build`, `test` and `push` is extracted to `/hooks/common`.
-3. The `build` script runs `run_build()` from `/hooks/common`.
-   This triggers either `/build-branches.sh`, `/build-latest.sh` or directly `/build.sh`.
-4. The `test` script just invokes `docker-compose` commands.
-5. The `push` script runs `run_build()` from `hooks/common` with a `--push-only` flag.
-   This causes the `build.sh` script to not re-build the Docker image, but just the just built image.
-
-The _Docker Tag_ configuration setting (`$DOCKER_TAG`) is only used to select the type (_release_, _prerelease_, _branches_) of the build in `hooks/common`.
-Because it has a different meaning in all the other build scripts, it is `unset` after it has served it's purpose.
-
-[overwrite]: https://docs.docker.com/docker-hub/builds/advanced/#override-build-test-or-push-commands

README.md

@@ -3,7 +3,7 @@
 [![GitHub release (latest by date)](https://img.shields.io/github/v/release/netbox-community/netbox-docker)][github-release]
 [![GitHub stars](https://img.shields.io/github/stars/netbox-community/netbox-docker)][github-stargazers]
 ![GitHub closed pull requests](https://img.shields.io/github/issues-pr-closed-raw/netbox-community/netbox-docker)
-![Docker Cloud Build Status](https://img.shields.io/docker/cloud/build/netboxcommunity/netbox)
+![Github release workflow](https://img.shields.io/github/workflow/status/netbox-community/netbox-docker/release)
 ![Docker Pulls](https://img.shields.io/docker/pulls/netboxcommunity/netbox)
 [![MicroBadger Layers](https://img.shields.io/microbadger/layers/netboxcommunity/netbox)][netbox-docker-microbadger]
 [![MicroBadger Size](https://img.shields.io/microbadger/image-size/netboxcommunity/netbox)][netbox-docker-microbadger]
@@ -21,7 +21,7 @@ Do you have any questions? Before opening an issue on Github, please join the [N
 [netbox-docker-github]: https://github.com/netbox-community/netbox-docker/
 [ntc-slack]: http://slack.networktocode.com/
 [netbox-docker-slack]: https://slack.com/app_redirect?channel=netbox-docker&team=T09LQ7E9E
-[netbox-docker-license]: https://github.com/netbox-community/netbox-docker/blob/master/LICENSE
+[netbox-docker-license]: https://github.com/netbox-community/netbox-docker/blob/release/LICENSE
 
 ## Docker Tags
 
@@ -124,31 +124,34 @@ docker-compose up -d
 
 From time to time it might become necessary to re-engineer the structure of this setup.
 Things like the `docker-compose.yml` file or your Kubernetes or OpenShift configurations have to be adjusted as a consequence.
+
 Since November 2019 each image built from this repo contains a `org.opencontainers.image.version` label.
 (The images contained labels since April 2018, although in November 2019 the labels' names changed.)
-You can check the label of your local image by running `docker inspect netboxcommunity/netbox:v2.7.1 --format "{{json .ContainerConfig.Labels}}"`.
+You can check the label of your local image by running `docker inspect netboxcommunity/netbox:v2.7.1 --format "{{json .Config.Labels}}"`.
 
 Please read [the release notes][releases] carefully when updating to a new image version.
 
 [releases]: https://github.com/netbox-community/netbox-docker/releases
 
-## Rebuilding & Publishing images
+## Rebuilding the Image
 
 `./build.sh` can be used to rebuild the Docker image. See `./build.sh --help` for more information.
 
-### Publishing Docker Images
+For more details on custom builds [consult our wiki][netbox-docker-wiki-build].
 
-New Docker images are built and published every 24h on the [Docker Build Infrastructure][docker-build-infra].
+[netbox-docker-wiki-build]: https://github.com/netbox-community/netbox-docker/wiki/Build
-`DOCKER_HUB.md` contains more information about the build infrastructure.
 
-[docker-build-infra]: https://hub.docker.com/r/netboxcommunity/netbox/builds/
+### Pre-made Docker Images
+
+New Docker images are built and published every 24h.
 
 ## Tests
 
-To run the tests coming with Netbox, use the `docker-compose.yml` file as such:
+We have a test script.
+It runs Netbox's own unit tests and ensures that all initializers work:
 
 ```bash
-docker-compose run netbox ./manage.py test
+IMAGE=netboxcommunity/netbox:latest ./test.sh
 ```
 
 ## About

VERSION

@@ -1 +1 @@
-0.21.0
+0.22.0

configuration/ldap_config.py

@@ -70,8 +70,7 @@ AUTH_LDAP_USER_FLAGS_BY_GROUP = {
 AUTH_LDAP_FIND_GROUP_PERMS = os.environ.get('AUTH_LDAP_FIND_GROUP_PERMS', 'True').lower() == 'true'
 
 # Cache groups for one hour to reduce LDAP traffic
-AUTH_LDAP_CACHE_GROUPS = os.environ.get('AUTH_LDAP_CACHE_GROUPS', 'True').lower() == 'true'
+AUTH_LDAP_CACHE_TIMEOUT = int(os.environ.get('AUTH_LDAP_CACHE_TIMEOUT', 3600))
-AUTH_LDAP_GROUP_CACHE_TIMEOUT = int(os.environ.get('AUTH_LDAP_GROUP_CACHE_TIMEOUT', 3600))
 
 # Populate the Django user from the LDAP directory.
 AUTH_LDAP_USER_ATTR_MAP = {

initializers/groups.yml

@@ -1,3 +1,15 @@
+## To list all permissions, run:
+## 
+## docker-compose run --rm --entrypoint /bin/bash netbox
+## $ ./manage.py migrate
+## $ ./manage.py shell
+## > from django.contrib.auth.models import Permission
+## > print('\n'.join([p.codename for p in Permission.objects.all()]))
+##
+## Permission lists support wildcards. See the examples below.
+##
+## Examples:
+
 # applications:
 #   users:
 #   - technical_user
@@ -8,9 +20,16 @@
 #   users:
 #   - writer
 #   permissions:
-#   - add_device
-#   - change_device
 #   - delete_device
-#   - add_virtualmachine
-#   - change_virtualmachine
 #   - delete_virtualmachine
+#   - add_*
+#   - change_*
+# vm_managers:
+#   permissions:
+#   - '*_virtualmachine'
+# device_managers:
+#   permissions:
+#   - '*device*'
+# creators:
+#   permissions:
+#   - add_*

initializers/users.yml

@@ -1,3 +1,15 @@
+## To list all permissions, run:
+## 
+## docker-compose run --rm --entrypoint /bin/bash netbox
+## $ ./manage.py migrate
+## $ ./manage.py shell
+## > from django.contrib.auth.models import Permission
+## > print('\n'.join([p.codename for p in Permission.objects.all()]))
+##
+## Permission lists support wildcards. See the examples below.
+##
+## Examples:
+
 # technical_user:
 #   api_token: 0123456789technicaluser789abcdef01234567 # must be looooong!
 # reader:
@@ -5,9 +17,7 @@
 # writer:
 #   password: writer
 #   permissions:
-#   - add_device
-#   - change_device
 #   - delete_device
-#   - add_virtualmachine
-#   - change_virtualmachine
 #   - delete_virtualmachine
+#   - add_*
+#   - change_*

startup_scripts/000_users.py

@@ -20,15 +20,23 @@ with file.open('r') as stream:
           username = username,
           password = user_details.get('password', 0) or User.objects.make_random_password)
 
-        print("👤 Created user ",username)
+        print("👤 Created user",username)
 
         if user_details.get('api_token', 0):
           Token.objects.create(user=user, key=user_details['api_token'])
 
-        user_permissions = user_details.get('permissions', [])
+        yaml_permissions = user_details.get('permissions', [])
-        if user_permissions:
+        if yaml_permissions:
-          user.user_permissions.clear()
+          subject = user.user_permissions
-          for permission_codename in user_details.get('permissions', []):
+          subject.clear()
-            for permission in Permission.objects.filter(codename=permission_codename):
+          for yaml_permission in yaml_permissions:
-              user.user_permissions.add(permission)
+            if '*' in yaml_permission:
-          user.save()
+              permission_filter = '^' + yaml_permission.replace('*','.*') + '$'
+              permissions = Permission.objects.filter(codename__iregex=permission_filter)
+              print("  ⚿ Granting", permissions.count(), "permissions matching '" + yaml_permission + "'")
+            else:
+              permissions = Permission.objects.filter(codename=yaml_permission)
+              print("  ⚿ Granting permission", yaml_permission)
+            
+            for permission in permissions:
+              subject.add(permission)

startup_scripts/010_groups.py

@@ -24,9 +24,18 @@ with file.open('r') as stream:
         if user:
           user.groups.add(group)
 
-      group_permissions = group_details.get('permissions', [])
+      yaml_permissions = group_details.get('permissions', [])
-      if group_permissions:
+      if yaml_permissions:
-        group.permissions.clear()
+        subject = group.permissions
-        for permission_codename in group_details.get('permissions', []):
+        subject.clear()
-          for permission in Permission.objects.filter(codename=permission_codename):
+        for yaml_permission in yaml_permissions:
-            group.permissions.add(permission)
+          if '*' in yaml_permission:
+            permission_filter = '^' + yaml_permission.replace('*','.*') + '$'
+            permissions = Permission.objects.filter(codename__iregex=permission_filter)
+            print("  ⚿ Granting", permissions.count(), "permissions matching '" + yaml_permission + "'")
+          else:
+            permissions = Permission.objects.filter(codename=yaml_permission)
+            print("  ⚿ Granting permission", yaml_permission)
+
+          for permission in permissions:
+            subject.add(permission)

startup_scripts/__main__.py

@@ -7,12 +7,12 @@ from os.path import dirname, abspath
 this_dir = dirname(abspath(__file__))
 
 def filename(f):
-    return f.name
+  return f.name
 
 with scandir(dirname(abspath(__file__))) as it:
-    for f in sorted(it, key = filename):
+  for f in sorted(it, key = filename):
-        if f.name.startswith('__') or not f.is_file():
+    if f.name.startswith('__') or not f.is_file():
-            continue
+      continue
       
-        print(f"Running {f.path}")
+    print(f"Running {f.path}")
-        runpy.run_path(f.path)
+    runpy.run_path(f.path)