Marcel Claassen
Alles over…
Marcel Claassen
Alles over…
Projects

P1 Monitor docker container for smart meters

door |Gepubliceerd |140 Reacties

A docker-based DSMR reader

This project is describing the porting of the P1-Monitor software running on Raspberry Pi to a container for running P1 monitor to be able to run it on docker or Kubernetes servers. The original software is maintained from ztatz.nl – Slimme meter monitor.

For those new to this subject, this software is as monitor for smart energy meters/slimme meters using the P1 port (a sort of serial port) using the DSMR 3 protocol. A data package, called a telegram, is sent every 10 seconds (interval may differ) that contains data on usage on electricity and gas. You can use that data to get real-time insight in your usage. P1 monitor is software that visualises the data using graphs en retain the information in databases.

Since I’m running a Linux server already I didn’t want to start using a dedicated Raspberry Pi board for monitoring my smart meter. But the software provided by ZTATZ is only available as a complete system image running on a Raspberry Pi. That’s why I started converting the software to a container version suitable for running on any system with the help of docker, docker compose, portainer or other orchestrator.
One other big benefit that I experienced is that I do not need to go through the extensive update process of exporting and importing data since everything is stored outside the container. Unless changes will be made to the databases of course but in that case you still use the export/import procedures.

The container image is available on Docker hub and I’m using GitHub to maintain the repository. The container version has the same functionality as the full version (although some options related to Operating System are not being used of course). Docker hub also holds the versions for other architectures so you can also run the container on ARM etc (such as Raspberry Pi) devices.

Connectors

Since I don’t have a serial port on my server (who does?) I used a standard USB to serial adaptor and modified a standard phone cable (for the RJ11 connector in the Smart meter) and connected it to the serial as below (note that RxD and TxD are swapped. Of course its even more easy to buy a standard ‘Smart meter USB cable‘ for 15 euro or so but it was weekend and I needed the cable now 🙂

P1-Monitor smart monitor cable connector

Install docker

For the ones not familiar with containers: Start with installing docker server version and docker-compose on your system. Using docker-compose it is easier to start, read and maintain a configuration. You can also install Docker Desktop but if you will not be using Docker for anything else its overkill. If you rather would like the use the Docker command-line options directly or integrate into a cluster examples can be found below.

Note: From July 2023 the docker-compose utility is not supported anymore. Instead use docker compose up -d.

Docker script

To start with docker-compose create a new directory and create a file named docker-compose.yml. Copy the contents of below docker-compose file example to that file and save it. (Note you keep the indentation with spaces!) If needed you can modify it using your own parameters such as the location of the data (in the example script in the subdirectory alldata). That subdirectory is for creating persistence; if the container is recreated or an update is available all data is preserved since it is being stored outside the container. If your server is rebooted the container will automatically be restarted.

Next you start your containerised P1-Monitor using the docker compose up -d command. A local network will be created and the container will be started in the background. After a short while (initially the container image needs to be downloaded from Docker hub) the P1-Monitor will be available on configured port. In below example it will be running on standard port 80 but you can easily change that by modifying the port mapping parameter from 80:80 to for instance 81:80 if you want the container to be available at port 8.

Similar is the devices section; it is mapping your local serial device to /dev/ttyUSB0 in the container. This is the device where you connected your smart meter.

Docker compose file example

version: '3.8'
services:

  p1monitor:
    hostname: p1mon
    image: mclaassen/p1mon
    #stop_grace_period: 1m
    ports:
      - 80:80
    volumes:
      - ./alldata/data:/p1mon/data
      - ./alldata/usbdisk:/p1mon/mnt/usb
      - ./alldata/mnt/ramdisk:/p1mon/mnt/ramdisk
    tmpfs:
      - /run
      - /tmp
    devices:
      - /dev/ttyUSB0:/dev/ttyUSB0
    restart: unless-stopped

Parameters

The docker-compose file supports below parameters:

  • ports
    mapping van ports from outside to inside. If you want to use port 81 to access your instance use the setting 81:80
  • volumes
    these are the directories being used to store the persistent data. In the example above subdirectory alldata will be automatically created (if started for the first time) and all collected information will be stored here.
  • devices
    mapping of your external to internal serial device. Adjustment is only needed in case you are using a different USB port on your host system.
  • stop_grace_period
    this option can be added to allow the stop script to be fully completed before the container is terminated. Without this parameter the container will be terminated in 10 seconds. In 10 seconds the complete stop is not executed (resulting e.g. that ramdisk data is not copied to data directory resulting in up to 15 minutes of data loss). The stop scripts from p1monitor need more time to complete and ramdisk data will be written to data directory. A period of approx. 1 minute should be sufficient.
    When using the stop_grace_period the container will be terminated when the stop script is completed or the defined period is timed out.
    Note: setting a too short period can cause databases to get corrupted! (if grace period ends during data backup)

Upgrade to latest version

If a new container is released (usually shortly after the original P1-Monitor software has been updated) you can stop the container instance using the docker compose down command. Do not delete any files but issue the command docker compose pull to retrieve the latest image from Docker hub. The pull command will download the latest image as specified in the docker-compose file. After the pull is complete start the container again using the docker compose up command. All historic data will be reused (if you do not change the docker-compose configuration file of course).

Note that new container versions will not automatically update at stopping and starting. The pull action must be preformed to retrieve a new version of the image.

Migrate from original P1-Monitor instance

To migrate from the original P1-Monitor (the one you had running on the dedicated Raspberry Pi) it is easiest to export the data. To do so use the menu Settings (Instellingen) and select In- Export. In your new instance use the same menu to import the extracted data again.

Additional docker compose parameters

Additional parameters can be passed to the container using the option environment. As example in above configuration adding the available parameters as shown below.

version: '3.8'
services:
  p1monitor:
    hostname: p1mon
    image: mclaassen/p1mon
    ports:
      - 80:80
    volumes:
      - ./alldata/data:/p1mon/data
      - ./alldata/usbdisk:/p1mon/mnt/usb
      - ./alldata/mnt/ramdisk:/p1mon/mnt/ramdisk
    tmpfs:
       - /run
       - /tmp
    devices:
       - /dev/ttyUSB0:/dev/ttyUSB0
    environment:
       - CRONTAB=/p1mon/data/crontab.txt
       - PROXYPATH=p1mon
       - SOCAT_CONF="-T60 pty,link=/dev/ttyUSB1,rawer,group-late=dialout,mode=660 tcp:192.168.1.200:5523,forever,interval=30"
    restart: unless-stopped

CRONTAB

Version 2.1.0 and later: Reference to a file containing additional crontab configuration settings. Note that used volume must be mapped correctly.

PROXYPATH

Since I would like access to the data also when not at home I added a reverse proxy option. By adding parameter PROXYPATH=p1mon the configuration in the container is adjusted so you can use the subdirectory p1mon as a path in your url. Your outside url would be something like https://myhomenetwork.nl/p1mon.
Don’t forget to protect your reverse proxy (e.g. username/password) as the P1-Monitor version is not build to be internet facing.

SOCAT_CONF

Some time back a few users requested to have socat implemented in the image for remote reading of the P1 data to a virtual serial port. The example parameter above is reading information from a remote serial reader at 192.168.1.200 port 5523. When starting the container port ttyUSB1 is created and will be used in P1Monitor to read the data. Logging is available in /var/log/socat.log. Please adjust the configuration as needed.

Depending on the remote device you’re using socat parameters may differ. Check the information from the supplier (or software vendor) for the correct parameters. Users have noted for instance that they needed to use raw or raw,echo 0 instead of rawer and modify group-late to group to get the connection up.

Note that the devices section is obsolete when using socat as the script is creating a virtual serial port itself.

GPIOZERO_PIN, PIGPIO_ADDR

Selecting pigpio as pinfactory for gpiozero allows using remote gpio, for example on a different raspberry. This is especially usefull when the docker container is running on a host that is not location where you want to use the gpio, for example to measure watermeter pulses. Adding GPIOZERO_PIN_FACTORY=pigpio and PIGPIO_ADDR=<somehost> to the docker-compose file is sufficient to allow p1monitor to use the gpios of a remote raspberry. The remote raspberry has to run the pigpiod.service, which is available in the pigpio raspbian/debian package.

Accessing GPIO ports

To get access for the GPIO pins on Raspberry Pi devices (for instance to read water meter information), add below modifications to your docker-compose script. Some users reported to have the privileged: true parameter added to the p1monitor section (just below the image tag) to get access to the GPIO ports. 

version: '3.8'
services:

  p1monitor:
    hostname: p1mon
    image: mclaassen/p1mon
    ports:
      - 80:80
    volumes:
      - ./alldata/data:/p1mon/data
      - ./alldata/usbdisk:/p1mon/mnt/usb
      - ./alldata/mnt/ramdisk:/p1mon/mnt/ramdisk
      - /sys:/sys
    tmpfs:
      - /run
      - /tmp
    devices:
      - /dev/ttyUSB0:/dev/ttyUSB0
      - /dev/gpiomem:/dev/gpiomem
    restart: unless-stopped

Using a ramdisk

The original software of P1 Monitor is using a ramdisk for writing to the databases. Reason is to avoid to many write actions to the SD cards (as they can become corrupted or something). Every now and then data from ramdisk is written to the filesystem. When running on a Linux server that is not a real issue as the file IO is not that much. If you still want to use a ramdisk you can use below modification in the docker-compose file. Remove the ramdisk from the volumes section and add it to the tmpfs section as shown below:

version: '3.8'
services:

  p1monitor:
    hostname: p1mon
    image: mclaassen/p1mon
    ports:
      - 81:80
    volumes:
      - ./alldata/data:/p1mon/data
      - ./alldata/usbdisk:/p1mon/mnt/usb
    tmpfs:
      - /run
      - /tmp
      - /p1mon/mnt/ramdisk
    devices:
      - "/dev/ttyUSB0:/dev/ttyUSB0"
    restart: unless-stopped

Storing logfiles outside container

When using the default configuration the logfiles are stored inside the container and visible via the P1-Monitor user interface or when logging into the container. If you want to have direct access to your logfiles you can store them outside the container by adding a volume such as ./alldata/log:/var/log/p1monitor in the volumes section.

Using docker with command-line options

Don’t want to use docker-compose? Of course you can also use the standard Docker command-line options. As noted earlier, more difficult to read but same functionality. This example, based on the first docker-compose example from above, will create a detached container named p1monitor listening on external http port 81, mapping /dev/ttyUSB0 to the same internal device and using/storing data in /home/user/alldata. You can modify and use below shell script to start using docker:

extport=81
datadir=/home/user/alldata
ttydev=/dev/ttyS0

docker run -d -p $extport:80\
       	-h p1mon\ 
        --name=p1monitor\
       	--tmpfs /tmp\
       	--tmpfs /run\
       	-v $datadir/data:/p1mon/data\
       	-v $datadir/usbdisk:/p1mon/mnt/usb\
       	-v $datadir/mnt/ramdisk:/p1mon/mnt/ramdisk\
       	--device $ttydev:/dev/ttyUSB0\
       	--restart=unless-stopped\
       	mclaassen/p1mon

Running in Kubernetes variants

Installation in a Kubernetes variant requires some specific configuration settings. Check page Running P1-Monitor in a Kubernetes environment for more information.

Version information

This is the list of changes for the different container versions. the latest tag (if you omit the tag latest will be used) will take you to the newest version.

  • 2.4.0f
    Fixed incompatible encryption library causing passwords not being stored
  • 2.4.0e
    Fixed MQTT backward compatibility
  • 2.4.0d
    Update base image to latest version of OS
  • 2.4.0c
    Small fixes and updated vulnerabilities
  • 2.4.0b
    Fix included for timezones and fixed vulnerabilities
  • 2.4.0a
    Fix included for phase scrolling
  • 2.4.0
    Update to latest released version 2.4.0
  • 2.3.0a
    Socat improvements and updates to latest patches
  • 2.3.0
    Update to latest released version 2.3.0
  • 2.2.0d
    Applied patch 3
  • 2.2.0c
    Applied patch 2, fixed issues on cron after restart
  • 2.2.0
    Update to latest released version 2.2.0 (integrated dynamic prices)
  • 2.1.0c
    Several fixes, updates on vulnerabilities and added support for dynamic energy prices
  • 2.1.0b
    Several fixes, updates on vulnerabilities and added support for pigpio (Using gpio remotely)
  • 2.1.0a
    Socat updates, p1monitor log files rotation option
  • 2.1.0
    Update to latest released version 2.1.0, additional crontab lines and logrotation of nginx logfiles
  • 2.0.0
    Update to latest released version 2.0.0
  • 1.7.0d
    Fixed critical PHP vulnerabilities and minor changes
  • 1.7.0c
    Fixed armv7 support
  • 1.7.0b
    Updated base image to latest version, added stop_grace_period support, fixed vulnerabilities and cleaned up scripts, no support for arm7 devices anymore
  • 1.7.0a
    Fixed issue where new installations could have a read-only configuration database
  • 1.7.0
    Minor changes and updated to latest released version 1.7.0
  • 1.6.0
    Updates to latest released version 1.6.0
  • 1.5.0d
    Fixed issue with rights on data directory and updated base image
  • 1.5.0c
    Fix issues related to ARM devices and GPIO port assignments
  • 1.5.0b
    Fixed minor issues, automated DISABLECPUTEMP variable (obsolete now)
  • 1.5.0a
    Updated system, completed socat support
  • 1.5.0 
    Updates to latest released version 1.5.0
  • 1.4.1b 
    Changed to running as p1mon, fixed issues, updated vulnerable packages
  • 1.4.1a 
    Fixed issue tariff status as described by author
  • 1.4.1 
    Based on the 1.4.1 image of ZTATZ. Also added reverse proxy capability
  • 1.4.0 
    Based on the withdrawn image of ZTATZ. Fixed 1 second issue. Image was withdrawn by the original author but due to the nature of docker container most bugs are not applicable here.
  • 1.3.1d 
    Fixed temperature error in VM using environment variable. Modified cron
  • 1.3.1c 
    Fixed import/export functions as well as missing ping utility
  • 1.3.1b 
    added health check function. Container will show health status and restart on failure
  • 1.3.1a 
    Updates fixing some vulnerabilities in python packages and dpkg
  • 1.3.1 
    Update to latest released version 1.3.1
  • 1.3.0 
    Update to latest released version 1.3.0
  • 1.2.0 
    Initial released version 1.2.0

More information

For more information on P1-Monitor such as latest updates, news and forum take a look at ZTATZ, GitHub, P1Monitor forum en op Docker Hub

Dit vind je misschien ook leuk

Een reactie plaatsen

Het e-mailadres wordt niet gepubliceerd. Vereiste velden zijn gemarkeerd met *

140 gedachten over “P1 Monitor docker container for smart meters”