A few weeks ago I set out to finally do something with a domain I'd been holding for the last few years, bitdash.io. As someone who both designs and codes user interfaces I wanted1 to have a place for displaying useful metrics related to the bitcoin network. However, since I had also been reading the logs since 2015, I knew enough to know that you don't just manalone and create something because you "just wanted to". So, when I joined the republic formerly known as TMSR in 2018 I joined as an apprentice. I saw mp-wp2 as an opportunity to learn while working. Consisting of a www app ultimately rendered in plain old HTML and CSS it presented itself as something I could contribute to given my existing skillset, but also something I could learn from (the PHP/MySQL parts were new to me, not to mention V).
Code was trimmed3, a feature was added, patches were produced, and in 2020 TMSR was closed. I had also at this time just moved to Costa Rica, and coincidentally, the world had decided to descend into full-blown authoritarianism—everywhere. And so I took a break from these things to study Spanish, take care of some much needed renovations at the ranch, and generally just enjoy the new country IRL. But one can only stay idle4 for so long, and eventually the desire to get back in front of a terminal and do something worthwhile exceeded my desire to relax.
So, back to bitdash.io. What data could I publish that would actually be useful to someone, and that wasn't already published on 1001 other bit-this, blockchain-that websites? Well, how about the one thing none of these other sites publish: anything related to TRB. I started with the simplest metric that I myself used to like to check now and then, the number of active TRB nodes on the network. The two sites that used to display this metric, however, no longer do. So I wrote a simple crawler5 and ended up with some interesting results. These findings gave me enough encouragement to want to continue with their publication, so that perhaps others may also see what I saw. It also gave me a clearer picture of what I hope to accomplish overall, and this made visible a deficit in my current stack, which is a place for both collaborating with other towards this goal and for sorting newcomers. And so, there is now #billymg on IRC, and logs.bitdash.io for those that wish to participate or just follow along. This article, with the exception of the former introduction, is about my small contribution to the code that renders those logs.
The First Patch
While working on the crawler I had already started to design a theme for the overall bitdash.io site and I figured the logs hosted on the same domain should have a similar visual aesthetic. So I configured asciilifeform's logotron and took a look at the HTML and CSS it was generating. There were a lot of things I didn't like6 so I first proceeded to rewrite/reorganize before writing my new theme. In doing so I also ended up refactoring a Python function that spit out an HTML string directly rather than simply passing data to an HTML template7. The first patch linked in this article contains these changes as well as a few minor functionality changes/additions. For those skimming, the changes in the first patch are as follows:
- A fairly comprehensive HTML/CSS refactor, as described above.
- Search queries less than Min_Query_Length (3 by default) characters redirect to the homepage rather than resulting in a 500 error.
- The ability to customize the logger's root path via
app_root
in the config8. - The ability to point to a different CSS file via
css_file
in the config. - A bunch more "bots" added to the default
bots
field in the config. - Some README updates with setup clarifications and a new reverse-chronological 'release_notes.txt' file.
Here is that gen_chanlist
function I mentioned:
1 | def gen_chanlist(selected_chan, show_all_chans=False): |
2 | # Get current time |
3 | now = datetime.now() |
4 | # Data for channel display : |
5 | chan_list = [] |
6 | chan_idx = 0 |
7 | for chan in Channels: |
8 | last_time = query_db( |
9 | '''select t, idx from loglines where chan=%s |
10 | and idx = (select max(idx) from loglines where chan=%s) ;''', |
11 | [chan, chan], one=True) |
12 | |
13 | last_time_txt = "" |
14 | last_time_url = "" |
15 | if last_time != None: |
16 | span = (now - last_time['t']) |
17 | days = span.days |
18 | |
19 | # Only add to the list if it should be visible, otherwise continue |
20 | if days > Days_Hide and chan != selected_chan and not show_all_chans: |
21 | continue |
22 | |
23 | hours = span.seconds/3600 |
24 | minutes = (span.seconds%3600)/60 |
25 | |
26 | if days != 0: |
27 | last_time_txt += '%dd ' % days |
28 | if hours != 0: |
29 | last_time_txt += '%dh ' % hours |
30 | if minutes != 0: |
31 | last_time_txt += '%dm' % minutes |
32 | |
33 | last_time_url = "{0}{1}{2}/{3}#{4}".format( |
34 | get_base(), |
35 | App_Root, |
36 | chan, |
37 | last_time['t'].strftime(Date_Short_Format), |
38 | last_time['idx']) |
39 | |
40 | chan_list.append({ 'name': chan }) |
41 | |
42 | chan_list[chan_idx]['last_time_url'] = last_time_url |
43 | chan_list[chan_idx]['last_time_txt'] = last_time_txt |
44 | chan_list[chan_idx]['chan_url'] = "{0}{1}{2}{3}".format( |
45 | get_base(), App_Root, chan, '/' if chan == Default_Chan else '') |
46 | |
47 | chan_idx += 1 |
48 | |
49 | return chan_list |
And in the HTML9 template:
1 | <table class='chan-list' align="center"> |
2 | <thead> |
3 | <tr> |
4 | {% for chan_item in chan_list %} |
5 | <th> |
6 | <a class='chan-link {% if chan_item.name == chan %}chan-link-active{% endif %}' |
7 | href='{{ chan_item.chan_url }}'><b>{{ chan_item.name }}</b></a> |
8 | </th> |
9 | {% endfor %} |
10 | </tr> |
11 | </thead> |
12 | <tbody> |
13 | <tr> |
14 | {% for chan_item in chan_list %} |
15 | <td> |
16 | <a class='chan-last-active-link' href='{{ chan_item.last_time_url }}'>{{ chan_item.last_time_txt }}</a> |
17 | </td> |
18 | {% endfor %} |
19 | </tr> |
20 | </tbody> |
21 | </table> |
Compared to before:
1 | def gen_chanlist(selected_chan, show_all_chans=False): |
2 | # Get current time |
3 | now = datetime.now() |
4 | # Data for channel display : |
5 | chan_tbl = {} |
6 | for chan in Channels: |
7 | chan_tbl[chan] = {} |
8 | chan_tbl[chan]['show'] = False |
9 | |
10 | chan_formed = chan |
11 | if chan == selected_chan: |
12 | chan_formed = "<span class='highlight'>" + chan + "</span>" |
13 | |
14 | chan_tbl[chan]['link'] = """<a href="{0}log/{1}"><b>{2}</b></a>""".format( |
15 | get_base(), chan, chan_formed) |
16 | |
17 | last_time = query_db( |
18 | '''select t, idx from loglines where chan=%s |
19 | and idx = (select max(idx) from loglines where chan=%s) ;''', |
20 | [chan, chan], one=True) |
21 | |
22 | last_time_txt = "" |
23 | time_field = "" |
24 | if last_time != None: |
25 | span = (now - last_time['t']) |
26 | days = span.days |
27 | hours = span.seconds/3600 |
28 | minutes = (span.seconds%3600)/60 |
29 | |
30 | if days != 0: |
31 | last_time_txt += '%dd ' % days |
32 | if hours != 0: |
33 | last_time_txt += '%dh ' % hours |
34 | if minutes != 0: |
35 | last_time_txt += '%dm' % minutes |
36 | |
37 | time_field = """<i><a href="{0}log/{1}/{2}#{3}">{4}</a></i>""".format( |
38 | get_base(), |
39 | chan, |
40 | last_time['t'].strftime(Date_Short_Format), |
41 | last_time['idx'], |
42 | last_time_txt) |
43 | |
44 | if (days <= Days_Hide) or (chan == selected_chan) or show_all_chans: |
45 | chan_tbl[chan]['show'] = True |
46 | |
47 | chan_tbl[chan]['time'] = time_field |
48 | |
49 | ## Generate channel selector bar : |
50 | s = """<table align="center" class="chantable"><tr>""" |
51 | for chan in Channels: |
52 | if chan_tbl[chan]['show']: |
53 | s += """<th>{0}</th>""".format(chan_tbl[chan]['link']) |
54 | s += "</tr><tr>" |
55 | ## Generate last-activ. links for above : |
56 | for chan in Channels: |
57 | if chan_tbl[chan]['show']: |
58 | s += """<td>{0}</td>""".format(chan_tbl[chan]['time']) |
59 | # wrap up: |
60 | s += "</tr></table>" |
61 | return s |
I personally find it much easier to read now. Also, when one is following the README, which before stated, "Adjust the three 'flask' templates in 'templates' subdir to give the desired look and feel for the www end", they can now change the look of their chan list as well, without have to modify any Python.
Now about that query. I did not fix that in either of these patches, but I did do some quick testing on my RK if anyone is curious about the results:
Current:
1 | SELECT t, idx FROM loglines WHERE chan='asciilifeform' AND idx = (SELECT max(idx) FROM loglines WHERE chan='asciilifeform'); |
2 | |
3 | t | idx |
4 | ---------------------------+--------- |
5 | 2021-05-25 18:51:44.88612 | 1037761 |
6 | (1 row) |
7 | |
8 | Time: 158.185 ms |
9 | |
10 | SELECT t, idx FROM loglines WHERE chan='trilema' AND idx = (SELECT max(idx) FROM loglines WHERE chan='trilema') |
11 | nsalog-# ; |
12 | t | idx |
13 | ----------------------------+--------- |
14 | 2020-03-13 08:47:33.022321 | 1959633 |
15 | (1 row) |
16 | |
17 | Time: 3.977 ms |
18 | |
19 | SELECT t, idx FROM loglines WHERE chan='ossasepia' AND idx = (SELECT max(idx) FROM loglines WHERE chan='ossasepia'); |
20 | t | idx |
21 | ----------------------------+--------- |
22 | 2020-09-22 03:57:31.860485 | 1028603 |
23 | (1 row) |
24 | |
25 | Time: 163.218 ms |
26 | |
27 | Total Time: 325.38 ms |
As a UNION:
1 | SELECT chan, t, idx FROM loglines WHERE chan='asciilifeform' AND idx = (SELECT max(idx) FROM loglines WHERE chan='asciilifeform') UNION |
2 | SELECT chan, t, idx FROM loglines WHERE chan='trilema' AND idx = (SELECT max(idx) FROM loglines WHERE chan='trilema') UNION |
3 | SELECT chan, t, idx FROM loglines WHERE chan='ossasepia' AND idx = (SELECT max(idx) FROM loglines WHERE chan='ossasepia'); |
4 | |
5 | chan | t | idx |
6 | ---------------+----------------------------+--------- |
7 | asciilifeform | 2021-05-25 18:51:44.88612 | 1037761 |
8 | ossasepia | 2020-09-22 03:57:31.860485 | 1028603 |
9 | trilema | 2020-03-13 08:47:33.022321 | 1959633 |
10 | (3 rows) |
11 | |
12 | Time: 292.233 ms |
As a UNION with knowledge of the last time queried:
1 | SELECT chan, t, idx FROM loglines WHERE chan='asciilifeform' AND idx = (SELECT max(idx) FROM loglines WHERE chan='asciilifeform' AND t > '2021-05-24') UNION |
2 | SELECT chan, t, idx FROM loglines WHERE chan='trilema' AND idx = (SELECT max(idx) FROM loglines WHERE chan='trilema' AND t > '2020-03-12') UNION |
3 | SELECT chan, t, idx FROM loglines WHERE chan='ossasepia' AND idx = (SELECT max(idx) FROM loglines WHERE chan='ossasepia' AND t > '2020-09-22'); |
4 | |
5 | chan | t | idx |
6 | ---------------+----------------------------+--------- |
7 | asciilifeform | 2021-05-25 18:51:44.88612 | 1037761 |
8 | ossasepia | 2020-09-22 03:57:31.860485 | 1028603 |
9 | trilema | 2020-03-13 08:47:33.022321 | 1959633 |
10 | (3 rows) |
11 | |
12 | Time: 167.390 ms |
Overall quite good, and likely even better in production with a more fine-grained time filter. I'll likely include this improvement in my next patch for the logotron if no one else beats me to it.
The Second Patch
The second patch is just a CSS file and a small snippet of HTML for the chan list. I simply could not reconcile the two chan list formats to achieve what I wanted to with my theme and also leave alf's theme intact even in terminal-based browsers. The problem wasn't even the table, that was easy to contort into a list with td { display: list-item }
. What ultimately made it unworkable was the fact that the original table used two separate table rows for the chan names and the last active times. Meaning the order of elements in the DOM was not chan1, chan1_ts, chan2, chan2_ts... but rather chan1, chan2...chan1_ts, chan2_ts... (you can witness this yourself when loading logs.nosuchlabs.com/log and tabbing through the channels at the top). At this point I gave up and decided to just include 'chan-nav-list.html' in the templates folder. By default it is unused, but one can point to it by changing one line in 'templates/layout.html', which is required if using the theme at 'static/bitdash.css'.
For those that would like to try these changes out, you will need these two patches and signatures, as well as the entire logotron tree from before (which is now mirrored here on this site).
frontend_updates.kv.vpatch
frontend_updates.kv.vpatch.billymg.sig
add_bitdash_theme.kv.vpatch
add_bitdash_theme.kv.vpatch.billymg.sig
There is also another logotron out there that I would like to try for myself and it is my plan to do so when I return to mp-wp work after first publishing the crawler.
- Why? I don't know, it just seemed like a "cool idea" (talk about towards purpose over from causes). [↩]
- The blogging platform of choice for many republican members, based off of a trimmed down fork of WordPress 2.7 [↩]
- A bit of an understatement, nearly 50% was hacked off. [↩]
- This "idle" of mine was still probably twice as demanding as what most chair warmers in various HR departments consider "busy". [↩]
- Using asciilifeform's Watchglass as my BTC protocol library [↩]
- Hard to read, HTML mixed in with Python, a mix of HTML and CSS styling, "!important" hacks for no reason, redundant or dead HTML/CSS [↩]
- This should generally only be done as a last resort or in very small/targeted doses as it defeats the purpose of code/markup separation and hurts overall readability. [↩]
- Note that 'app_root' is somewhat of a misnomer as search, for example, appears at '$base_url/log-search', rather than '$base_url/$app_root/search'. This is something I would like to change but it would result in a lot of broken links in the existing log data that would have to be bulk updated and so I left it alone for now. [↩]
- Jinja [↩]
Looking sharp, thanks for putting a bot in my chan. Having a logger inspires the channel's use. I'll get my own logger up soon and maybe even make my own theme now that you and alf have made it easy. Incoming one-to-one human to logbot ratio.
Side note: did you change your wp header on the default theme so you're able to adjust the width? IIRC this theme by default uses a goddamn image to get the blue background behind the header text.
Excellent, and glad to hear it had a positive effect.
Yes, my theme on this blog is the default wp theme with some minor modifications, the first of which was to remove any image backgrounds in favor of plain CSS. The default theme not only uses an image for the header background but also a y-repeating image for the entire body. Very easy to remove. Feel free to grab the CSS file served here, diff against the default, and take what you like.