I've been using asciilifeform's logotron to power my IRC logger for a while now. The setup consists of an IRC bot and a Flask-based web app, both connected to a Postgres database. I run it on a simple Rockchip machine alongside my Bitcoin network crawler, which also uses Postgres. For some reason every couple of weeks¹ the bot would lose its DB connection and have to be manually restarted² in order to continue eating log lines. Same with the crawler. Potentially there's a performance issue somewhere, causing the connection to timeout and close in some cases. I know it's not the logotron itself, since asciilifeform ran the same code on the same box for almost a year and never had the DB connection issue, but perhaps both apps sharing the same small RK is just too much for it to handle at times.

In any case, it was a problem for me, as well as others—and I figured it couldn't hurt for the bot to have auto-db-reconnect functionality—so I came up with a small patch to remedy the issue. Below is the tidied up version of what I currently have running.

With this running locally I was able to bring down the DB, send a few IRC messages, restart the DB, and have the messages that were sent while the DB was offline recorded once the DB came back up. I'm going to give it a few weeks in production to see how it fares. If it does indeed solve the issue (and not cause any others) I'll package it as a vpatch and publish it here along with the rest of the logotron tree.

1. Sometimes longer, but also sometimes much shorter. I suspect it has to do with the usage/load of the sites. [↩]
2. This happened often enough that I even wrote a small script to pull from asciilifeform's channel logs when mine fell behind:

   #!/bin/bash
   
   batch_size=500
   
   start_line=$1
   end_line=$2
   chan_url=$3
   results_file=$4
   
   if [ $1 -ge $2 ]; then
           echo "end line must be greater than the start line"
           exit 1
   fi
   
   num_lines=$(( $end_line - $start_line ))
   num_batches=$(( $num_lines / ($batch_size + 1) + 1 ))
   last_batch_size=$(( $num_lines % ($batch_size + 1) ))
   
   echo "starting at ${start_line}, ending at ${end_line}, for a total of ${num_lines} lines"
   echo "will download from ${chan_url}?istart=${start_line}&iend=${end_line} in ${num_batches} batch(es). last batch will be ${last_batch_size}"
   
   batch=1
   while [ $start_line -lt $end_line ]; do
           if [ $batch -eq $num_batches ]; then
                   cur_end=$(( $start_line + $last_batch_size ))
           else
                   cur_end=$(( $start_line + $batch_size ))
           fi
   
           echo "pulling ${start_line} - ${cur_end}"
           curl -s "${chan_url}?istart=${start_line}&iend=${cur_end}" >> $results_file
   
           start_line=$(( $cur_end + 1 ))
           batch=$(( batch + 1 ))
   done
   

   [↩]

This entry was posted on Thursday, January 6th, 2022 at 6:48 p.m. and is filed under bitdash.io, logotron, postgres. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.