GitLab with Docker : Fixing “Error: PG::ConnectionBad” or “DETAIL: The data directory was initialized by PostgreSQL version 9.6, which is not compatible with this version 11.7.”

An issue I had where I could not find any fix online. The closest help I could find was this blog post : https://gotanbl.com/foss/how-update-gitlab-in-docker/, but there are multiple mistakes in the article, and the author is not reachable to fix them, so I’ll put the main trivia here…

The root of the issue (in my case) is that updating GitLab (with docker at least) is quite cumbersome. If you run it almost every week, then you can always upgrade to “latest”. If you do it from time to time, you have to upgrade from minor to latest minor then to first major, then to latest minor, then first major etc… And there is no way to do that automatically. So sometimes, running the “usual command” won’t work.

The problem which leads to this error is that the Postgresql database version will only be updated on some versions. If you skip the right one, then you’ll never update and all subsequent updates will break…

Worst, you’ll end up being told to run some commands to do this and that… However the problem is that the docker container will die as it fails to start. So you won’t be able to enter those commands.

The solution

First, note your current version:

sudo docket exec -it gitlab bash
cat /opt/gitlab/version-manifest.txt |grep gitlab-ce|awk '{print $2}'

Then, stop and remove the container. It’s safe, as the real files, db, etc are kept in the $GITLAB_HOME:

sudo docker exec -t gitlab gitlab-backup create
sudo docker stop gitlab
sudo docker rm gitlab

Basically, you’ll have to follow a specific upgrade path that can be found at : https://docs.gitlab.com/ce/update/#upgrade-paths

At the time of writing, this is the path:

8.11.x -> 8.12.0 -> 8.17.7 -> 9.5.10 -> 10.8.7 -> 11.11.8 -> 12.0.12 -> 12.1.17 -> 12.10.14 -> 13.0.14 -> 13.1.11 - > 13.x (latest)

So if your version is 11.10, you’ll have to upgrade at 11.11.8, and continue up to the latest.

To update to a version, do the following.

Verify the command to run the container matches what you used to install GitLab in the first place, you should re-use exactly the same command, only the last $VERSION should change:

export GITLAB_HOME=/srv/gitlab
sudo docker run --detach --hostname gitlab.tombarbette.be --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.tombarbette.be/'; gitlab_rails['gitlab_shell_ssh_port'] = 2022; " --publish 2443:443 --publish 2080:80 --publish 2022:22 --name gitlab --restart always --volume $GITLAB_HOME/config:/etc/gitlab --volume $GITLAB_HOME/logs:/var/log/gitlab --volume $GITLAB_HOME/data:/var/opt/gitlab gitlab/gitlab-ce:$VERSION

The $VERSION should be the version with “-ce.0”, so for instance 11.11.8-ce.0, this is the docker container version that can be found at https://hub.docker.com/r/gitlab/gitlab-ce/tags?page=1&ordering=last_updated

Normally, when you launch that command the update for postgresql should be done automatically. If somehow you start hearing complaints, you can force the upgrade to the version XXX with:

gitlab-ctl pg-upgrade -v XXX

Where XXX is the database version.

After launching a specific version, you have to wait for GitLab to completely start, to be sure all migration was terminated.

If in troubles, you might want to check the logs with :

sudo docker logs -f gitlab

Typically the logs in $GITLAB_HOME starts to be meaningful when this problem is fixed and gitlab completely started, so it was not helpful for me.

So at this point, go back to the version list above and advance one by one…

It may seem crazy, but now to avoid that you have no choices than updating every weeks… Or you’ll have to play with versions again…

Now, I run a cron script that will save and update GitLab every week. Anyway, GitLab is an horror memory-wise and slows down with time. So removing and re-adding the container every week is actually helpful…

#!/bin/bash
sudo docker exec -t gitlab gitlab-backup create
sudo docker stop gitlab
sudo docker rm gitlab
export GITLAB_HOME=/srv/gitlab
sudo docker run --detach --hostname gitlab.tombarbette.be --env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.tombarbette.be/'; gitlab_rails['gitlab_shell_ssh_port'] = 2022; " --publish 2443:443 --publish 2080:80 --publish 2022:22 --name gitlab --restart always --volume $GITLAB_HOME/config:/etc/gitlab --volume $GITLAB_HOME/logs:/var/log/gitlab --volume $GITLAB_HOME/data:/var/opt/gitlab gitlab/gitlab-ce:latest

A poster of our latest work, CrossRSS, a Stateless CPU-Aware Datacenter Load-Balancer

Today we will present a poster of our latest work, at CoNEXT’20 : CrossRSS! CrossRSS is a load-balancer that spreads the load uniformly even inside the servers. It uses knowledge of the dispatching done inside the servers, RSS, to purposely select less-loaded cores without any server modification, or inter-core communications on the server. Learn more by watching the short video!

The poster session will be held on the 4th of December, 2:30 CET on the Mozilla VR Hub

Extended Abstract ; Hub ; Video ; Poster-As-Slides

Dynamic DNS with OVH

It may not be a clear thing, but OVH allows to have your own Dynamic DNS if you rent a domain name, surely a better thing than the weird paid website from dyndns.org. I will explain how to handle the update with Linux using ddclient.

On the manager

Connect to https://www.ovh.com/manager/web/#/configuration/domain/ , select your domain name, and create a new dynhost with the button on the right.

Enter a sub-domain name such as “mydyn” (.tombarbette.be), and add the actual IP for now, or just 8.8.8.8 for the time being.

Then it is not finished, you have to create a login that will be able to update that dns entry. Select the second button to handle accesses and create a new login.

Select a login, probably the name of the subdomain, the subdomain itself, and a password.

On the server

sudo apt install ddclient

Then edit /etc/ddclient.conf

protocol=dyndns2
use=web,web=checkip.dyndns.com
server=www.ovh.com
login=tombarbette.be-mydns
password='password'
mydns.tombarbette.be

Just do “sudo ddclient” to update once then “sudo service ddclient restart” to get it updated automatically.

May this be helpful to someone, personally I just forget it all the time so I wanted to leave a post-it somewhere.

PROXIMUS_AUTO_FON automatic connexion on linux using wpa_supplicant

If you understand this title, you don’t need more explanation :

/etc/network/interfaces
auto wlan1
iface wlan1 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

/etc/wpa_supplicant/wpa_supplicant.conf
ctrl_interface=/var/run/wpa_supplicant

network={
ssid="PROXIMUS_AUTO_FON"
scan_ssid=1
key_mgmt=WPA-EAP
eap=TTLS
identity="LOGIN@proximusfon.be"
password="PASS1234"
phase2="auth=MSCHAPV2"
}

Some may ask why some people would want to do that… I’m now using Voo, but I use my parent’s FON login when voo crash. My current project is towards aggregating the two links by load balancing, or at least have some kind of automatic failover. The more interesting part would be to switch to “FON only” when I reach my 100Gb limit…

Install and share the Canon Pixma MX395 Scanner with Sane

Found a Pixma MX395 at 27€ yesterday… It’s quite easy to find the Canon debian package to install the printer (use these one and not the included) and “scangearmp” which is the specific tool from Canon to scan, but it is not standard, and do not allow to share your scanner on the network through SANE.

The current version of sane do not support that printer, so you’ll need to use an updated one. Do :

sudo add-apt-repository ppa:rolfbensch/sane-git
sudo apt-get install sane sane-utils libsane

And it’s up !

scangearmp -L should show your scanner :
scanimage -L < ~ [14:04:02] device `v4l:/dev/video0' is a Noname USB2.0 Camera virtual device device `pixma:04A91766_21F9AD' is a CANON Canon PIXMA MX390 Series multi-function peripheral

Also edit /etc/sane.d/saned.cong to add the network subnet which can access the scanner :
10.0.0.0/24
[2a02:578:3fe:8139::]/64

For me. Do not forget the IPv6 address, of course 😉

Then on your client, install sane and edit /etc/sane.d/net.conf to add the server address :
10.0.0.1

And if you run scanimage -L on your client you should now see the remote scanner :
scanimage -L
device `v4l:/dev/video0' is a Noname USB2.0 UVC HD Webcam virtual device
device `net:10.0.0.1:v4l:/dev/video0' is a Noname USB2.0 Camera virtual device
device `net:10.0.0.1:pixma:04A91766_21F9AD' is a CANON Canon PIXMA MX390 Series multi-function peripheral

Proximus BBOX 3 in bridge mode on Linux (ubuntu,debian,raspbian,centos,fedora,…)

Using bridge mode allows you to get a public IP address on one computer (which can serve as a router) behind your modem. This allows you to know your public IP address without using a third-party service, and control more finely all your routing parameters inside your own Linux-based router (this tutorial) or a better router than the BBOX’s one (which is not that bad, now that I can, unfortunately, compare with VOO…). We’ll call “the router” the device you want to use behind the modem for clarity.

The bridge mode of the Proximus BBOX 3 is quite interesting. You connect normally to your BBOX using DHCP and will get a locally routable address (i.e 192.168…), but you can use PPP over Ethernet (PPPoE) to get a virtual interface inside your router. This virtual “ppp” interface will have a public IP address, and packets will flow IN and OUT the internet through that interface.

Proximus allows you to therefore maintain 2 PPP connections, one established in the BBOX (also used for the TV), and the other inside your router. It also means your home gets 2 IPv4 addresses.

I prefer that mode to the VOO one, where the external IP address is given by DHCP to only one host in the LAN, the first device to connect to the router using DHCP (dangerous and prone to configuration errors…). Same and independently for IPv6 using DHCPv6. While Proximus not only gives you an IPv6 address but also a /64 prefix via PPPoE to get a direct connection without using a crappy NAT to all your PCs. For IPv6, Proximus is much simpler than setting up an independent DHCPv6 client which gives back the v6 prefix to your LAN side. The second downside is that VOO must use ugly hacks to allow connection to the box as there is no “modem internal network” anymore. You can access your modem at the normally-illegal 192.168.100.1 address as this is on the “public web” space from the router perspective. Moreover, it seems that the modem stops responding to DHCP requests from time to time, losing connectivity… VOO bridge mode is definitively not good… But this may be a temporary bug. I did not observe this anymore…

The bridge part

Edit /etc/network/interfaces to add the following lines , assuming that eth0 is the interface used to connect to your BBOX.
auto dsl-provider
iface dsl-provider inet ppp
pre-up /bin/ip link set eth0 up
provider dsl-provider

Install pppoe with sudo apt-get install pppoe on ubuntu/debian or sudo yum install pppoe centos/fedora

Then create a file named /etc/ppp/peers/dsl-provider and add the following lines :

noipdefault
defaultroute
replacedefaultroute
hide-password
noauth
persist
mtu 1492
plugin rp-pppoe.so eth0
user "fc0123456@skynet"
usepeerdns

Then edit the file /etc/ppp/chap-secrets and add the line :
"fc012345@skynet" * "password"

If you lost your skynet credentials (personally, I just never received them), you can change them online on MyProximus. You’ll have to reboot your modem so it receives automatically the new credentials.

And that’s all, you can reboot or do a sudo pon dsl-provider and you’ll have a new interface with a public IPv4 and a /64 IPv6.

The router part

To give connectivity in IPv4 for your hosts and use your Linux host as a router, you’ll have to NAT. But you can delegate your IPv6 range and give public IPv6 addresses to all your PCs using SLAAC! Remember to also install a firewall…

To do so, install radvd and add in /etc/radvd.conf (if br0 is the interface connected to your internal network) :

interface br0
{
AdvSendAdvert on;
prefix ::/64
{
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
};
RDNSS 2001:4860:4860::8888 2001:4860:4860::8844
{
# AdvRDNSSLifetime 3600;
};
};

Then do a sudo radvd restart and that’s it.

The RDNSS line give the address of Google’s public DNS to your host. We could use Proximus’ one, but I don’t have the address on hand.

Note that I write this tuto only now but I did this month ago, I may have forgotten some steps, do not hesitate to contact me 😉

Creating a dynamic and redundant array with LVM and MDADM

RAID5 allows to create an array of N+1 drives where N is the number of drives which will contain real data. The last drive will be used to store parity about the other drives (in practice, the parity information is stored by chunks across all drives and not only on one drive). RAID 5 allows to loose any of the drive without loosing the data thanks to the parity drive, and has a cheaper cost than RAID 1 where the usable data will be N/ instead of N-1.

MDADM is the tool of predilection to build a RAID5 drive. Given 3 disks, the command to build a raid 5 array is :

mdadm --create /dev/md0 --level=5 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1

Problem is, RAID5 drives are not easily splittable/shrinkable/resizable, the operation is complex and must be done offline. The solution is to use LVM on top of MDADM to build a big volume group which will be “protected” by RAID5 allowing to make dynamic paritions on it :

pvcreate /dev/md0
vgcreate group0 /dev/md0

And then create multiple, online-resizeable partitions with :

lvcreate /dev/group0 -n system -L 10G
mkfs.ext4 /dev/mapper/group0-system

lvcreate /dev/group0 -n home -L 50G
mkfs.ext4 /dev/mapper/group0-home

To resize a partition, one can do :

lvresize /dev/mapper/group0-home -L +10G
resize2fs /dev/mapper/group0-home

Which will add 10G to the partition, and resize it. It will work even with the system partition, without needing any reboot.

 

Enable Wifi N access point with hostapd

I use an odroid (a rasberry-pi like mini-pc but more powerfull) as a Wifi access point for my smartphone and my camera since quite a long time. I forgot that my USB Wifi dongle was compatible with Wifi N (only on 2.4Ghz), so my hostapd config file was :

interface=wlan3
ssid=Barbette-Chambre
hw_mode=g
channel=11
bridge=br0
wpa=2
wpa_passphrase=YOURPASSPHRASE
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
rsn_pairwise=CCMP
wpa_ptk_rekey=600

Here is the speed result with iperf :

[ 4] local 10.0.0.44 port 5001 connected with 10.0.0.175 port 48727
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.2 sec 18.4 MBytes 15.1 Mbits/sec

Normaly, this should be 56Mbits/s, but we know wifi is crap…

And to enable Wifi N :

interface=wlan3
ssid=Barbette-Chambre
hw_mode=g   #Yes, this is not an error. Wifi N builds on top of G ;)
channel=11
bridge=br0
ieee80211n=1
wmm_enabled=1
country_code=BE
ht_capab=[HT20][HT40][SHORT-GI-20][SHORT-GI-40]
ieee80211d=1
wpa=2
wpa_passphrase=YOURPASSPHRASE
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
rsn_pairwise=CCMP
wpa_ptk_rekey=600

 

And the speed result is now :

[ 4] local 10.0.0.44 port 5001 connected with 10.0.0.175 port 48754
[ 4] 0.0-10.1 sec 30.6 MBytes 25.4 Mbits/sec

Better, but still not the 150Mbits/s of wifi N… But it’s better !

Limit rate of mail sent by motion detection daemon (motiond) for webcam surveillance

Maybe you use the “on_picture_save” to send yourself an e-mail with motiond when your webcam detect an  intruder. But when someone enters, you’ll receive a lot of pictures. And if it was just you forgetting to stop the detection before entering, your smartphone will become a christmas tree for nothing…

I just published my last program “limitrate” on Github :

https://github.com/tbarbette/limitrate

This programs allows you to limit the rate of launching a certain command. Instead of launching

/usr/bin/program argument1 argument2

call :

limitrate 30 “/usr/bin/program % argument2” argument1

To launch only this command at maximum once every 30 seconds. If you call multiple times the last line in 30 seconds, limitrate will run the command only after 30 seconds replacing the “%” by the concatenation of all “argument 1”. For example to send an alert e-mail containing files (for example webcam picture) at max every 30 seconds :

10:24:30 : limitrate 30 "echo \"Alert!\" | mailx % destination@mail.com" "-A /path/to/file1"
--> Send a mail with file1
10:24:32 : limitrate 30 "echo \"Alert!\" | mailx % destination@mail.com" "-A /path/to/file2"
10:24:38 : limitrate 30 "echo \"Alert!\" | mailx % destination@mail.com" "-A /path/to/file3"
10:24:41 : limitrate 30 "echo \"Alert!\" | mailx % destination@mail.com" "-A /path/to/file4"
10:25:05 : limitrate 30 "echo \"Alert!\" | mailx % destination@mail.com" "-A /path/to/file5"
--> Send a mail with file 2,3,4 and 5

Example configuration for motion detection daemon (Camera surveillance) :

on_picture_save /usr/local/bin/limitrate 15 “echo \”See attachment below.\” | mailx -s ‘Motion in the saloon !’ %destination@tombarbette.be” “-A %f”

When an intrusion is detected, you’ll receive a first e-mail with one image. 15 seconds later you’ll receive all the new images in only one mail, and so on.

ZSH : Open terminal where you left, for each session

There is some snippets for ZSH configuration which allow you to re-open the session in the folder where it was last closed available on the web. The problem is that you often launch 3 sessions at the same time, work on them and then quit/reboot/loose SSH connections/… So you will re-log 3 sessions which will start in the same last opened folder.

I propose a version allowing to keep the last folder per-session. Each ZSH session receive a number and write the current folder in a per-session file. When you open a new session it opens the file number associated to the session number.

 

Add somewhere in .zshrc :

mkdir -p ~/.cwd/
session_num=`pgrep zsh | wc -l`
function cd() {
builtin cd "$@";
echo "$PWD" > ~/.cwd/$session_num
echo "$PWD" > ~/.cwd/last
}
export cd
function cwd() {
if [ -e ~/.cwd/$session_num ] ; then
cd "$(cat ~/.cwd/$session_num)"
else
cd "$(cat ~/.cwd/last)"
fi
echo "This is session #$session_num"
}

And at the bottom of the file :

cwd