Vim Plugin for Web Publishing

2026-03-09

Frank Siebert

I wrote lengthily how I moved away from wordpress to write articles in mediawiki and publish via git hook to my then new designed web site. Now its time to report the sunset of my mediawiki. I moved on to Vim and GitLab flavored Markdown.

> [!note] > **Update yyyy-MM-dd:** Note text as it applies

| %25% Header1 | %25% Header2 | %50% Header3 | |:-------------|:-------------|:----------------------| | text 1 | text 2 | Much longer text here |

| Header1 | Header2 | Header3 | |:-------------|:-------------|:----------------------| | text 1 | text a | belongs to 1.a | | ^ | text b | belongs to 1.b | | text 2 | ^ | belongs to 2.b |

I'll not go into the details of the migration. It's enough to mention that I extracted all wiki content as mediawiki files and converted those to markdown via Pandoc.

Those mediawiki files, which where my article source files, where converted the same way.

No, this time I will restrict myself to describe the authoring and publishing plugin I created for Vim. Another article might follow to describe the implementation of the deployment via git hook.

The appearance of the web site stays the same, and my design goals for the site as well.

• No JavaScript, at least not for the portal, only for the content, if it does require it.
• No resources linked into the site, which are loaded from 3rd-party servers.

Sure, my content has links to other sites, as I'm used to provide references in my articles. But you have to click on these consciously to be taken to the referenced page. The visitor is in control.

Table of Content

Plugin Short Description

It's a bit more, but this is the core functionality:

• Render markdown into HTML, applying the semantic HTML structure, the style and, if applicable, media links, as they show up in the published result.
• Hand the rendered HTML and its assets over to the deployment staging area of my site.

Some assets are generated together with the HTML, the QR-Code which contains the link to the article and the PDF containing the same article as the HTML.

If an audio is recorded by me to provide the article as podcast, that is linked into the HTML as well.

Those steps where done before by software launched via git-hook during commit, which means that my deployment process became much leaner now. But that is part of the deployment to the web-site, and I plan to write a separate article about that.

Meta Data

There are two kinds of meta data, which, due to the lack of a better place, have to be inserted into the markdown file. To place this meta data information into the markdown file, I decided to use XML/HTML comments like:

<!-- pdf -->

The plugin provides a list of meta data comments to insert the selected one at the current cursor position.

Control Data

Shall a PDF be created? Do I want to have a table of content, and where would I like to have that? Where shall the references be placed? Where are additional page-breaks required in the PDF? Which language is the article in?

While writing mainly in German, there are sometimes exceptions to that. This article, for example, is in English, and will therefore not be shown as part of the "idee"-portal, but of the "concept"-portal. This makes the language information in my case control data.

It is, however, also article meta data.

Article Meta Data

Who is the author? Under which Licence is the content provided?

As long as I write all articles myself, always using the creative-commons zero licence, I could just hard code this information. And a lot things, which should be customizable, are currently not. But I decided to enable guest articles with different content licence. At least in principle that function is there, I would need to add some information about this other content licence the first time it is used.

Workflow and File System Structure

Lets first take a look at the target file system structure.

frank@Asimov:~/projects/idee/website$ tree -d -L 1 . ├── archive ├── article ├── audio ├── css ├── env ├── files ├── image ├── js ├── legal ├── MathJax -> SimpleMathJax/resources/MathJax/es5 ├── pdf ├── portal ├── qrcode ├── SimpleMathJax └── sitemap

Articles, audios, files, images, JavaScripts, PDFs and QR-Codes are placed in separate folders next to next of each other.

At one hand we might want to re-use an image or JavaScript already existing, at the other hand we want to have easy access to those files we are working with, and want to avoid searching for these in a long list of files day by day.

To facilitate both options, the authoring happens in a separate folder, one per article currently in creation, with the asset-folders beneath and a symbolic link to the idee folder, by which existing assets can be referenced in the markdown file.

That looks as follows:

File System Structure

frank@Asimov:~/projects/writing/1-vim-plugin-idee$ tree  . ├── audio ├── files ├── idee -> /home/frank/projects/idee ├── image ├── js ├── pdf ├── qrcode └── vim-plugin-for-web-publishing.md

To create this setup for authoring again and again by hand would be too much work. It is done by the plugin command IdeeFolders .

Workflow

The workflow is quite lean and basic:

1. create article folder in your preferred location
2. vim your-markdown-file.md
3. write the article
4. call command IdeeFolders
5. create and link assets
6. add meta data via command IdeeMeta or
7. call command IdeeDisplay to render HTML (opens the page in Firefox)
8. re-iterate steps 5 to 7 until everything fits
9. call command IdeePublish

The command IdeePublish copies the markdown, the HTML and all assets to the idee folder structure and adjusts the relative links used during authoring to fit to the new location.

Step 9 is the handover to the deployment of the article.

Feature List

As shown in the workflow description, the plugin features 4 commands:

• IdeeFolders creates the asset folder and the symbolic link to the idee website project.
• IdeeMeta provides a list of meta data comments
• IdeeDisplay creates the HTML and opens it in the web-browser
• IdeePublish hands the article and its assets over to the deployment

Plugin Folder Structure

This, I confess, took some time to figure out. Where do I need to place what and how to I make sure that it is loaded when a markdown file is edited?

As soon as it works, it seems to be simple.

frank@Asimov:~/projects/writing/1-vim-plugin-idee$ tree -L 5 ~/.vim/pack/idee/ /home/frank/.vim/pack/idee/ └── start  └── plugin  ├── ftplugin  │  └── markdown_idee.vim  └── python3  └── idee  ├── display.py  ├── folders.py  ├── __init__.py  ├── publish.py  └── __pycache__

The folder pack probably stands for package, I do not know. If so, then I have a plugin package named idee , with a folder start which is, as I understand, searched during the start of vim, whether applicable plugins are found below.

Below folder plugin there is the ftplugin folder, where the code resides, which makes the python functions available via new vim commands.

For whatever reason the python code needs to be placed below a python3 folder or it is not found by vim.

File Type Plugin markdown_idee.vim

There is a binding naming convention for file type plugins, and if you do not adhere to it, it is not loaded when a file of that type is edited.

The name has to start with the file type, as it is known to vim. That can be followed by an underline and whatever you like to write afterwards. I stuck with the plugin name, which is also my web-site project name.

markdown_idee.vim

py3 import idee  function! markdown_idee#meta_comments()  let l:comments = [  \ '<!-- author: --> ',  \ '<!-- article-licence: cc0 --> ',  \ '<!-- en-US --> ',  \ '<!-- page-break --> ',  \ '<!-- pdf --> ',  \ '<!-- references --> ',  \ '<!-- toc --> '  \ ]   function! s:meta_insert(id, result) closure  if a:result > 0 && a:result <= len(l:comments)  execute "normal! i" . l:comments[a:result - 1]  endif  endfunction   let winid = popup_menu(l:comments, #{  \ callback: 's:meta_insert'  \ }) endfunction  command IdeeFolders :py3 idee.folders.folders(vim.eval("expand('%')")) command IdeeDisplay :py3 idee.display.display(vim.eval("expand('%')")) command IdeeMeta call markdown_idee#meta_comments() command IdeePublish :py3 idee.publish.publish(vim.eval("expand('%')"))

The documentation of vim still refers to the command python , but that doesn't work, it needs to be py3 . The file type plugin imports the python package idee with its modules.

The meta_comments() function is also defined in vim-script, defining a nested callback function to insert the selected comment into the current editor buffer at the current cursor position.

Oh, yes, the function shows a popup menu with the comments to choose from.

After loading the python package and defining the function meta_comments() the file type plugin binds the python functions and the vim-script function to new vim commands.

The current markdown file, the file the author is working in when using the commands, is represented by % . Python, called via py3 doesn't know that and wouldn't know what to do with that percent sign. Therefore this placeholder for the current file name needs to be expanded into the file name using the python function vim.eval() provided by the vim developers.

Module initialization

The python modules are imported via the __init__.py of the package.

__init__.py

""" The package 'idee' provides functions to be used via command in vim during the editing of markdown files, which are meant to be published as articles on the idee-website or the concept-website.  Functions provided are:  idee.folders():  - Creates the folder structure for assets  - Creates a link to the idee/concept website project  idee.display()  - Creates idee/concept website style HTML from the markdown file.  - Creates a corresponding PDF file, if requested.  - Shows the result in Firefox  idee.publish()  - Does the handover to the idee/concept website project for deployment """ import os from pathlib import Path import pkg_resources  import idee.folders import idee.display import idee.publish

IdeeFolders

The module folders creates subfolders below the folder the current markdown file is in.

Function folders()

folders.py - folders()

""" folders(filepath) creates folders for idee website assets. It creates also a symbolic link to the idee project to make existing assets available.  These folders are:  - files - arbitrary linked files  - image - image files  - js - javascript files for interactive content  - pdf - article pdf  - qrcode - article qrcode  - audio - recording of the article  For the target project idee a symbolic link is created  - ln -s idee  @date: 2026-02-19 """  import os import string from pathlib import Path  def folders(filepath: string):  """  Make the folder of the given markdown file a workfolder.  This is done by creating the subfolders for possible assets.   Returns  -------  Feedback Message via stdout  """   dlist = ["files", "image", "js", "pdf", "qrcode", "audio"]   inpath = Path(filepath)  workdirpath = inpath.parents[0]  workdirpath = workdirpath.resolve()   # create symbolic link to idee project  idee = workdirpath / "idee"  if not idee.exists():  os.symlink(Path("/home/frank/projects/idee").resolve(), idee)   for d in dlist:  d = workdirpath / d  if not d.exists():  d.mkdir()

IdeeDisplay

The module display creates HTML, on request via meta data comment also PDF, and opens the created page in Firefox. It makes sense to look at this module part by part.

display.py - imports and debugging

#!/user/bin/python3 """ Generate and Display HTML in idee-website style from github flavored markdown.   To debug the function display(), start this executable python file with  the markdown file as parameter.   TODO: Check alternatives to soft deprecated module getopt.  @date: 2026-02-05 """  import sys import os import subprocess import string from pathlib import Path from datetime import datetime import re import copy import getopt  from selenium import webdriver from selenium.webdriver.chrome.options import Options  from bs4 import BeautifulSoup from bs4 import Comment from bs4.builder._htmlparser import HTMLParserTreeBuilder from weasyprint import HTML from weasyprint import CSS import qrcode ... if __name__ == "__main__":   MARKDOWNFILE = None   try:  opts, args = getopt.getopt(sys.argv[1:], ["o"])   except getopt.GetoptError:  print("No Parameter given")  sys.exit(2)   if len(args) == 0:  print("No Parameter given")  sys.exit(2)   MARKDOWNFILE = args[0]  if not MARKDOWNFILE:  print("No Parameter given")  sys.exit(2)   display(Path(MARKDOWNFILE))   sys.exit(0)

The code shows the imports and the __main__ part used to call the display function from terminal for e.g. debug purposes. The ellipsis shown in the middle of this code snipped stands for all the function definitions in between, which in the following will be shown one by one.

Function display()

The display function uses Pandoc to create the initial HTML from the markdown file. Afterwards the module BeautifulSoup is used to adjust the HTML to the style used for the idee/concept portal.

display.py - display() - part 1

def display(filepath: string):  """  Creates an html web page frm the markdown file.   The html web page is fully flavored for that site,  including the optional linked in pdf and audio versions  and content licence information.   Returns  -------  None.  """   mdpath = Path(filepath)  workdirpath = mdpath.parents[0]   # check that this a designated workdirectory  idee = workdirpath / "idee"  if not idee.exists():  print("Use first IdeeFolders to designate this location as"  " workdirectory, or use the PreVim-Plugin instead."  )  return   htmlpath = workdirpath / mdpath.stem  htmlpath = htmlpath.with_suffix(".html")   # To enable --toc, the parameter -s (standalone) needs to be set.  # This parameter leads to the generation of an html header with  # some meta tags.   # The TOC is created as <nav id="TOC"> tag,   # Own meta data lines need be injected and  # the toc needs to be moved to the correct location if specified,  # or removed, if specified.  htmltext = subprocess.run( [ "pandoc", "-s",  # create table of content  "--toc", "--toc-depth=5",  # use MathJax  "--mathjax" +  "=./idee/website/MathJax/tex-chtml.js",  # github flavor markdown  "-f", "gfm",  # html as output format  "-t", "html",  # input file  "-i", mdpath  # don't use stdout, return the result  ],  capture_output=True,  check=True)

The function first checks the existence of the symbolic link to the idee website project. If that link does not exist, then obviously the current directory had not been prepared to serve as authoring directory for an idee/concept article.

Pandoc is called to convert from GitLab flavored markdown to HTML and is told to process also MathTex statements like:

From "The Cosmological Constant as Event Horizon" ^{( 1 )}

\[\begin{equation} \nabla_{\mu}g^{\mu} = \frac{d\mathsf{\Theta}}{d\tau} + \frac{1}{3}\mathsf{\Theta}^{2} = R_{\mu\nu}u^{\mu}u^{\nu} = \mathsf{\Lambda} - 4\pi G(\rho + 3p) \end{equation}\]

The HTML-comments in the markdown are preserved during conversion to HTML. With this meta information and with the target HTML structure in mind, the returned HTML is processed further.

display.py - display() - part 2

 html_doc = htmltext.stdout.decode("utf-8")   # do some things the soup does not want to do  html_doc = html_doc.replace("<body>",  "<body><main><article><header></header>")  html_doc = html_doc.replace("</body>", "</article></main></body>")   builder = HTMLParserTreeBuilder()  soup = BeautifulSoup(html_doc, builder=builder)   # remove script added by pandoc  scripts = soup.find_all("script")  for script in scripts:  script.decompose()   style(soup)  urnmeta(soup, mdpath.stem)  images(soup, workdirpath)  language(soup)  title(soup)  dates(soup, workdirpath, mdpath.stem)  tables(soup)  article_author(soup)  article_qrcode(soup, workdirpath, mdpath.stem)  article_licence(soup)  article_audio(soup, workdirpath, mdpath.stem)  movetoc(soup)  footnotes(soup)  # keep this as last step  article_pdf(soup, workdirpath, mdpath.stem, htmlpath)   html_doc = soup.prettify()   with open(htmlpath, 'w', encoding='utf-8') as outfile:  print(html_doc, file=outfile)  outfile.flush()  outfile.close()   print(f'wrote file {htmlpath}')   subprocess.run(["firefox", htmlpath], capture_output=False, check=True)

That's it. First I sneak in the target HTML structure, having a main tag containing the article tag, which contains a header as first element to place some header information about the article.

Afterwards I create a BeautifulSoup and then remove the scripts placed by Pandoc, since I want to have HTML free of scripts as long as its not absolutely required.

Then I have a number of functions running on that soup to add and modify some details based on the meta data and the assets found, into which we will look next in detail.

Function style()

The function style() adds everything to the soup, which I consider to be part of the portals style.

display.py - style()

def style(soup):  """  Show some personal style.  """  # inject the generator meta information.  # one exists already  tag = soup.find("meta", attrs={"name": "generator"})  tag.attrs.update({"name": "generator", "content": "vim, pandoc, idee"})   head = soup.find("head")  # inject ghost message from Terry Prachett  # http://www.gnuterrypratchett.com/  meta = soup.new_tag("meta")  meta.attrs.update({"http-equiv": "X-Clacks-Overhead",  "content": "GNU Terry Pratchett"})  head.insert(6, meta)   # pandoc creates a style tag. if --mathml option is set  tag = soup.find("style")  if tag:  tag.decompose()  # my own style  # inject stylesheet link <link rel="stylesheet"  # href="../website/css/fs.css"/>  link = soup.new_tag("link")  link.attrs.update({"rel": "stylesheet", "href": "./idee/website/css/fs.css"})  head.insert(7, link)   # pandoc creates <div class="note"><div class="title">...</div></div>  # for > [!nete] and similar for > [!important]. The titles have to go.  tags = soup.find_all("div", class_="title")  for tag in tags:  tag.decompose()   # favicon  # href="../image/favicon.ico" rel="icon" type="image/x-icon"  link = soup.new_tag("link")  link.attrs.update({"rel": "icon", "href": "./idee/website/image/favicon.ico",  "type": "image/x-icon"  })  head.insert(8, link)   math = soup.find("span", class_="math")  if math:  # behind title tag, before MathJax loading  script = soup.new_tag("script")  script.attrs.update({"type": "text/javascript"})  script.append("window.MathJax={tex: {tags: 'span'}};") # instead of all  head.insert(9, script)   script = soup.new_tag("script")  script.attrs.update({"type": "text/javascript",   "src": "./idee/website/MathJax/tex-chtml.js"})  head.insert(9, script)

Part of style is the honoring the software used to create the web-page in the generator meta data. I used vim, Pandoc and now my own software, named like my web-site and vim-plugin simply 'idee'.

Terry Prattchet gets honored in the X-Clacks-Overhead ^{( 2 )} , which is not very effectful, but my web-server honors him as well in the HTTP Header, which makes the X-Clacks Browser-Extension signal his presence in the overhead.

Not less important is the link to the CSS-stylesheet, the web-sites favicon icon, and, only in case span tags of type 'math' are found, the script for MathJax.

Function urnmeta()

In my previous implementation the title used in mediawiki became the file name of the mediawiki file, and I had to compute an URN, an unique resource name, to be used as HTML file name.

That meant very long file names and, well, the mediawiki file names even had spaces. That mess is over now, at least for new articles.

display.py - urnmeta()

def urnmeta(soup, urn):  """  The name of the markdown file becomes urn for the web.  Filenames are all lowercase with '-' instead of spaces,  and roman letters only (no german umlaute).   I got tired of mixed case filenames with spaces anyhow during  my use of my mediawiki extract tool.  """  # meta data  head = soup.find("head")  meta = soup.new_tag("meta")  meta.attrs.update({  "property": "article:urn",  "content": urn})  head.insert(6, meta)

As URN parameter mdpath.stem is used, that forces me to take care while I name my markdown file, that I adhere to the self set rules.

Yes, I should probably add a check for that, and probably I will. And probably I should retire that meta data entry, since the information exists first in the file name and later in the URL.

Function images()

This function looks at all img tags and, if no relative path is given to the image, looks in which of the two possible image folders the image can be found.

That means, that I can use the image file name in the markdown without any path.

Images used in the header, especially the qrcode image, are in different folders. Therefore this function has to be called early and not be moved to later process stages.

display.py - images()

def images(soup, workpath):  """  Run early, before we place pictures in the header.   Inspect image source path. Keep realtive paths alone,  lookup image location if only the image filename is given.   PARAMETER  soup:  The html soup  workpath:   The location of the md-file  """  imgs = soup.find_all("img")  for image in imgs:  src = image.attrs.get("src")  if src and "./" not in src:  imagepath = None  imgp = None  for imgp in ["image", "idee/website/image"]:  imgp = workpath / imgp / src  if imgp.exists():  imagepath = imgp  break   if imagepath:  src = f"./{os.path.relpath(imagepath, workpath)}"  image.attrs.update({"src": src})   if src:  # make image anchors with target _blank to open in separate tab  anchor = soup.new_tag("a")  anchor.attrs.update({"href": src, "target": "_blank"})  image.insert_after(anchor)  anchor.append(image)

Function iscomment()

Time to introduce a very small helper function to find all comments we placed in the markdown file.

display.py - iscomment()

def iscomment(elem):  """  Helper function to search for comments   source:  https://www.tutorialspoint.com/beautiful_soup/beautiful_soup_find_all_comments.htm  """  return isinstance(elem, Comment)

Function language()

The language function maintains all language specific aspects. That's for one the xml-lang information in the HTML tag and the language meta tag, but it's also the HTML comment with the information, whether the idee web-site portal or the concept web-site portal has to be included for the SSI header injection done be the web-server.

If no language information was provided in the markdown file, it defaults to German.

display.py - language()

def language(soup):  """  Check for <!-- en-US --> language information comment.  If not, assume de-DE.   Populate the language attributes at the html tag and insert  the include comment for the portal.  """   # look for a language comment  comments = soup.find_all(string=iscomment)  c = None  for comment in comments:  if comment in ' en-US ':  c = comment  break  if c:  lang = str(c).strip()  c.decompose()  else:  lang = "de-DE"   # SSI header injection is a function of the language  body = soup.find("body")  if lang.startswith("de"):  c = Comment('# include file="/portal/idee-header.html" ')  else:  c = Comment('# include file="/portal/concept-header.html" ')   body.insert(0, c)   # inject language informationi  html = soup.find("html")  html.attrs.update({"lang": lang})  html.attrs.update({"xml:lang": lang})   # and 2 lines meta information  # <meta content="de-DE" property="og:locale"/>  # <meta content="Idee" property="og:site_name"/>  head = soup.find("head")   meta = soup.new_tag("meta")  meta.attrs.update({  "property": "og:locale",  "content": lang})  head.insert(6, meta)   meta = soup.new_tag("meta")  meta.attrs.update({  "property": "og:site_name",  "content": "idee" if lang.startswith("de") else "concept"})  head.insert(6, meta)

Function title()

The title is also inserted in multiple locations, in a title tag in the header and in a meta tag in the HTML head.

display.py - title()

def title(soup):  """  Move the title to the location we want it to have and fill also the  corresponding meta data tag with the title information.  """  h1 = soup.find("h1")  t = None  if h1:  t = h1.text  h1.decompose()  else:  t = "No title found"   # target header structure: check div exists, otherwise create  # <header><h1></h1><div><time></time><address></address></div></header>  header = soup.find("header")  h1 = soup.new_tag("h1")  h1.string = t  header.insert(0, h1)   # meta data  head = soup.find("head")  meta = soup.new_tag("meta")  meta.attrs.update({  "property": "og:title",  "content": t})  head.insert(6, meta)   titletag = soup.find("title")  titletag.string = t

Since Pandoc already created an h1 tag, it needs just to be moved into the desired location.

Function dates()

Only two dates are saved in the HTML, the published date, which is put into the meta data and visibly into the article header, and the last modified date.

As I canceled my central meta data storage, I have to look whether an older version of the article exists and read the correct publishing date from that article.

display.py - dates()

def dates(soup, workpath, urn):  """  Create the meta and also the visual date tags.  - article:modified_time: current local time  - article_published_time: current local time if article is published first  time, otherwise the time is taken from the published article.  """  head = soup.find("head")  d = datetime.now().isoformat()   meta = soup.new_tag("meta")  meta.attrs.update({  "property": "article:modified_time",  "content": d[:19]})  head.insert(6, meta)   htmlpath = workpath /Path(f"idee/website/article/{urn}").with_suffix(".html")  if htmlpath.exists():  builder = HTMLParserTreeBuilder()  with open(htmlpath, 'r', encoding='utf-8') as pf:  published_html = pf.read()  pf.close()  published_soup = BeautifulSoup(published_html, builder=builder)  timetag = published_soup.find("time", attrs={"pubdate": "true"})  if timetag:  d = timetag["datetime"]  else:  print("Warning: Published article has no pubdate.")   meta = soup.new_tag("meta")  meta.attrs.update({  "property": "article:published_time",  "content": d[:19]})  head.insert(6, meta)   # target header structure: check div exists, otherwise create  # <header><h1></h1><div><time></time><address></address></div></header>   header = soup.find("header")  div = header.find("div")   if not div:  div = soup.new_tag("div")  header.append(div)   t = soup.new_tag("time")  div.insert(0, t)  t.string = d[:10]  t.attrs.update({"datetime": d[:19]})  # probably deprecated by itemprop alternative  t.attrs.update({"pubdate": "true"})

Function tables()

Enables table column width control via %dd% entry in front of the respective column title. Numbers are always interpretated as percentage.

Enables the creation of rowspan for individual cells, by marking the cells to join via ^ .

display.py - tables()

def tables(soup):  """  1. Look in <th> tags for %dd% entries at the start of the text.  Use dd for style: with:dd%  2. Look in <td> tags for lone ^ entries.  Update the rowspan in the respective <td> tag of the predecessing row.  Dispose the ^ tag.  3. Look in <td> tags for lone < entries.  Update the columnspan in the predecessing <td> tag.  Dispose the < tags. (implementation pending)  """   # 1  tags = soup.find_all("th")  for tag in tags:  text = tag.string  match = re.match(r'^%(\d{2}%)', text)  width = match.group(1) if match else None  if width:  s = tag.get("style")  if s:  s = f"{s} width:{width};"  else:  s = f"width:{width}"  tag.attrs.update({"style": s})  text = text[4:].strip() # fits for one and two digits  tag.string = text   # 2  # to prevent index becoming a moving target, iterate backwards  while True:  tags = soup.find_all("td", string='^')  if not tags:  break  tag = tags[-1]  if tag.parent: # we decompose tags as we go, therefore we have to check  sibl = tag.parent.find_all("td")  ## obviously search for index does compare strings, not identity  index = len(sibl) - 1 - sibl[::-1].index(tag) # backward search  pstag = tag.parent.find_previous_sibling("tr") # parent search tag  rowspan = 2  tag.decompose()  while pstag and pstag.find_all("td")[index].string in '^':  rowspan += 1  pstag.find_all("td")[index].decompose()  pstag = pstag.find_previous_sibling("tr")  if pstag:  pstag.find_all("td")[index].attrs.update({"rowspan": f"{rowspan}"})

Function article_author()

The author of the article is written into the meta data and visibly into the article header.

If a comment exists, naming an author, this information is used. Otherwise my name is assumed to be correct.

display.py - dates()

def article_author(soup):  """  Check for author information provided as comment.  If not, assume Frank Siebert.   Populate the author information in the soup.  """  comments = soup.find_all(string=iscomment)  tag = None  a = None  for comment in comments:  if 'author:' in comment.text:  tag = comment  break  if tag:  a = tag.text.split(':')[1].strip()  tag.decompose()  else:  a = "Frank Siebert"   head = soup.find("head")  meta = soup.new_tag("meta")  meta.attrs.update({  "property": "article:author",  "content": a})  head.insert(6, meta)   # target header structure: check div exists, otherwise create  # <header><h1></h1><div><time></time><address></address></div></header>  header = soup.find("header")  div = header.find("div")   if not div:  div = soup.new_tag("div")  header.append(div)   address = soup.new_tag("address")  address.string = a  div.append(address)

Function article_qrcode()

Every article shows the QR-Code of its URL. This makes it easy to open the article also on a mobile, if you first see it on a computer screen or on paper.

The future article location is fully known by its URN.

display.py - article_qrcode()

def article_qrcode(soup, workpath, urn):  """  Run after author()  PARAMETER  soup:  workpath: The location of the md-file  urn: unique resource name of the article   Based on thr urn we know the url the web-page will have on  idee.frank-siebert.de, and url the qrcode image will have.   For this url we create the qrcode image, if it does not already exist, and  show it in the html header.  """  qrpath = None  qrp = None  for qrp in ["idee/website/qrcode", "qrcode"]:  qrp = workpath / qrp / urn  qrp = qrp.with_suffix(".png")  if qrp.exists():  qrpath = qrp  break   if not qrpath: # create it  qrpath = qrp  docurl = "https://idee.frank-siebert.de/article/" + urn + ".html"  image = qrcode.make(data=docurl)  image.save(qrpath)   # write into the soup, first figure in the second div of the header  header = soup.find("header")  div = header.find_all("div")  if len(div) == 0:  print("call article_qrcode() after author()")  elif len(div) == 1:  div = soup.new_tag("div")  header.append(div)  else:  div = div[1]   figure = soup.new_tag("figure")  div.append(figure)   caption = soup.new_tag("figcaption")  figure.insert(0, caption)   anchor = soup.new_tag("a")  anchor.attrs.update({"href": f"./{os.path.relpath(qrpath, workpath)}"})  figure.insert(0, anchor)   img = soup.new_tag("img")  img.attrs.update({"width": "150px", "height": "150px"})  img.attrs.update({"src": f"./{os.path.relpath(qrpath, workpath)}"})  img.attrs.update({"alt": "QR Code"})  anchor.insert(0, img)

Function article_licence()

This function looks, whether a comment with licence information exists, and includes then the respective licence information into the article header.

A dictionary with a respective configuration exists currently only in the python module and contains only two entries for the "creative commons zero" licence. This is also the licence used, if no other licence information is found.

Module variable LICENCES

LICENCES = {  "cc0": [  {  "href":  "./idee/website/legal/creative-commons-cc0-1-0-universal.html",  "alt": "Creative Commons",  "img": "./idee/website/image/CC-Icon.png"  },  {  "href":  "./idee/website/legal/creative-commons-cc0-1-0-universal.html",  "alt": "Zero",  "img": "./idee/website/image/CC0-Icon.png"  }  ]  }

display.py - article_licence()

def article_licence(soup):  """  Insert the licence information, if requested by the author via  <!-- article-licence: cc0 -->. Other licences might be added as required   TODO: A variant of this method could search for <!-- licence: xy -->  informations to replace them with licence information for 3d-party content.  """  comments = soup.find_all(string=iscomment)  c = None  for comment in comments:  if 'article-licence:' in comment:  c = comment  break   if not c:  return   c = c.split(":")[1].strip()   licences = LICENCES.get(c)  if not licences:  print(f"No matching licence information found for {c}.")  return   header = soup.find("header")  div = header.find_all("div")  if len(div) < 2:  print("Call article_licence() after article_qrcode()")  return   div = div[1]   for licence in licences:  anchor = soup.new_tag("a")  div.append(anchor)  anchor.attrs.update({"href": licence.get("href")})  img = soup.new_tag("img")  # The following scaling is for the PDF  # In the browser the CSS overwrites this scaling:  # newtag.attrs.update({"width": "28px", "height": "28px"})  img.attrs.update({"src": licence.get("img")})  img.attrs.update({"alt": licence.get("alt")})  anchor.append(img)

Function article_audio()

This function looks whether a recording exists, sharing the same name as the HTML, just with extension .mp3 . It looks into the two folders designated for audio files.

If the audio is found, it is added to the article header.

display.py - article_audio()

def article_audio(soup, workpath, urn):  """  Run after article_licence()  PARAMETER  soup:  workpath: The location of the md-file  urn: unique resource name of the article   Based on thr urn we know the url the audio file has to have.   """  audiopath = None  aup = None  for aup in ["audio", "idee/website/audio"]:  aup = workpath / aup / urn  aup = aup.with_suffix(".mp3")  if aup.exists():  audiopath = aup  break   if not audiopath:  return   header = soup.find("header")  div = header.find_all("div")  if len(div) < 2:  print("call article_audio() after article_licence()")  return   div = div[1]  figure = soup.new_tag("figure")  div.append(figure)  audio = soup.new_tag("audio")  figure.append(audio)  audio.attrs.update({"accesskey": "a",  "type": "audio/mp3",  "preload": "none",  "controls": "true",  "src": f"./{os.path.relpath(audiopath, workpath)}"})

Function movetoc()

If a comment for the TOC exists, the method moves the TOC generated by Pandoc to the location of the comment, replacing it.

If no comment for the TOC exists, then no TOC was desired and the TOC is removed.

The highest level of the TOC, containing the article title, is removed. The highest level used for headlines inside the articles is <h2> .

display.py - movetoc()

def movetoc(soup):  """  Move the TOC to the correct location  The author will set <!-- toc --> if he wants a TOC,  and he will do it in the location, where it should be.  """  toc = soup.find("nav", id='TOC')  # Eliminate the article title from the toc  ul1 = toc.find("ul")  ul2 = ul1.find("ul")  ul1.replace_with(ul2)  tag = None  comments = soup.find_all(string=iscomment)  for comment in comments:  if comment in ' toc ':  tag = comment  break  if tag:  tag.replace_with(toc)  else:  toc.decompose()

Function footnotes()

The footnotes function moves the references section created by Pandoc to the location of the references comment.

More a style matter, but best done here, is the replacement of the default back-reference symbol with a symbol that indeed points backwards.

display.py - footnotes()

def footnotes(soup):  """  Move footnotes to the place of the authors choice  given by <!-- references -->  Change back reference symbol to reference back and <CR>  """  # use a better symbol for backreferences  tags = soup.find_all("a", string='↩︎')  for tag in tags:  tag.clear()  tag.append('↑')   # Footnotes are generated as section <section class="footnotes"   # role="doc-endnotes"> - Search Section and use it to replace references comment.  fnotes = soup.find("section", class_="footnotes")  if fnotes:  tag = soup.find(string=iscomment, text='references')  if tag:  tag.replace_with(fnotes)

article_pdf()

At last the question is, whether PDF-creation is requested. If this is the case, then the PDF and the link to it in the HTML page are created.

display.py - article_pdf()

def article_pdf(soup, workpath, urn, htmlpath):  """  Checks for <!-- pdf -->, which is the request to create PDF.   Applies some changes to the html:  - figure for the audio is senseless in PDF  - relative URL's need to become absolute.  - footnotes need written links to make sense on paper  - creates the pdf  - places the link to the PDF into the soup  """  comments = soup.find_all(string=iscomment)  c = None  for comment in comments:  if 'pdf' in comment:  c = comment  break   if not c:  return   csspath = Path(r"/home/frank/projects/idee/website/css/fspdf.css")   workpath.resolve()  pdfpath = workpath / "pdf" / urn  pdfpath = pdfpath.with_suffix(".pdf")   pdfsoup = pdf_soup(soup)  page_break(pdfsoup)   html_doc = pdfsoup.prettify()   # weasy can't render MathTex, since it doesn't run the required JavaScript  # let chromium hender MathTex to HTML and create PDF from that  math = soup.find("span", class_="math")  if math:  html_doc = pdf_math(html_doc, workpath, htmlpath)   weasy_html = HTML(string=html_doc, base_url=str(workpath))  bpdf = weasy_html.write_pdf(stylesheets=[CSS(filename=str(csspath))])  with open(pdfpath, 'wb') as outfile:  outfile.write(bpdf)  outfile.flush()  os.fsync(outfile)  outfile.close()   # insert pdf symbol and link into the original soup  link_pdf(soup, urn)

As can be seen in the code, pdf creation requires a number of steps, which will be detailed below.

pdf_soup()

The HTML needs a bit adjustment before the PDF is created. To avoid messing up the to be published HTML, a separate BeautifulSoup is created.

display.py - pdf_soup()

def pdf_soup(soup):  """  Creates a separate soup for the PDF, because there some slight  differences.   First of all we need to honor additional software in the generator  information. Then PDF is not suited for audio and we need to remove that.   We also change all links into absolute links to make sure, that the  generator finds all images and cascading style sheets.    Returns:  soup  """  # create an own soup for pdf  html = soup.prettify()   builder = HTMLParserTreeBuilder()  pdfsoup = BeautifulSoup(html, builder=builder)   # honor weasy as generator, and also chromium if we render math  math = soup.find("span", class_="math")  meta = pdfsoup.find("meta", attrs={"name": "generator"})  generators = meta["content"]  if math:  meta.attrs.update({"name": "generator",  "content": f"{generators}, chromium, weasy"})  else:  meta.attrs.update({"name": "generator",  "content": f"{generators}, weasy"})   # The article header  article = pdfsoup.find("article")  header = article.find("header")   tag = header.find("figure", attrs={"class": "audio"})  if tag:  tag.decompose()   # We need to change relative paths to own articles into absolute  # paths.  rhref = re.compile(r"^\.\.\/")  anchors = pdfsoup.find_all("a", href=rhref)  for anchor in anchors:  url = rhref.sub("https://idee.frank-siebert.de/article/",  anchor["href"])  anchor.attrs.update({"href": url})   # On paper we need complete written URLs  rhref = re.compile(r"^http.*")  tag = pdfsoup.find("section", class_="footnotes")  if tag:  anchors = tag.find_all("a", href=rhref)  for anchor in anchors:  url = anchor["href"]  anchor.parent.append(pdfsoup.new_tag("br"))  anchor.parent.append(url)  return pdfsoup

page_break()

Creating PDF means also, that automatic page breaks might simply look awful. It might e.g. let a headline stand alone at the bottom of one page, to start the chapter at the next page.

As mentioned, page-break comments can be included anywhere in the markdown file to take control of page-breaks in the PDF.

display.py - article_pdf()

def page_break(soup):  """  Find <!-- page-break --> comments set by the author and replace them  with <div style="page-break-before: always;"> to control page break  positions in the pdf  """  comments = soup.find_all(string=iscomment)  for comment in comments:  if 'page-break' in comment:  pb = soup.new_tag("div")  pb.attrs.update({"style": "page-break-before: always"})  comment.replace_with(pb)  # if the page-break is in the table, split table  if pb.parent.name == 'td':  tr = pb.parent.parent # should be "tr"  tbody = tr.parent  table = tbody.parent  thead = table.find("thead")  newthead = copy.copy(thead)  newtable = soup.new_tag("table")  table.insert_after(pb)  pb.insert_after(newtable)  newtable.append(newthead)  tbody = soup.new_tag("tbody")  newtable.append(tbody)  while (tr_next := tr.find_next_sibling("tr")) is not None:  tbody.append(tr_next)  tbody.insert(0, tr)

pdf_math()

I found the resulting PDF most to my satisfaction using weasy to create it. I tried it first with Pandoc, and most probably the results could look the same with that tool, but for now my decision stands with weasy.

The only problem: Weasy does not support JavaScript, which is required to use MathJax to render MathTex into MathML.

Multiple solutions can be imagined to solve this issue, including running JavaScript directly from python, but I thought it a genius idea to use chromium to render MathML.

display.py - pdf_math()

def pdf_math(html_doc, workpath, htmlpath):  """  Use Chromium to render MathTex into MathML via MathJax, since weasy does  not support javascript.   TODO: Implement load and script control for webdriver to know exactly when  MathJax scripting finished, to get rid of the sleep.   Returns;  string - html_doc with rendered MathML  """  # we use the same filename as for the final html  # that way we do not need to delete the temporary file  with open(htmlpath, 'w', encoding='utf-8') as outfile:  print(html_doc, file=outfile)  outfile.flush()  os.fsync(outfile)  outfile.close()   # with webdriver  chrome_options = Options()  chrome_service = \  webdriver.ChromeService(executable_path="/usr/bin/chromedriver")  chrome_options.add_argument("--headless")  chrome_options.add_argument("--enable-javascript")  chrome_options.add_argument("--window.navigator.webdriver=False")  chrome_options.add_argument("--useAutomationExtension=False")  chrome_options.add_argument("--disable-blink-features=AutomationControlled")  chrome_options.add_argument("--user-agent=Mozilla/5.0 (X11; Linux x86_64)" \  " AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 " \  "Safari/537.36")  chrome_options.add_experimental_option("excludeSwitches",  ["enable-automation"])  browser = webdriver.Chrome(options=chrome_options, service=chrome_service)   temp = workpath / htmlpath  temp = temp.resolve()  browser.get( f"file://{temp}" )  # wait till the page finished changing  time.sleep(3)   test = ""  while True:  html_doc = browser.page_source  if html_doc in test:  break  test = html_doc   return html_doc

I consider the usage of chromium as a workaround, and I always tell everyone, that one workaround incubates the next. And indeed, you see the next workaround directly afterwards as time.sleep(3) .

As I'm not religious about workarounds, only cautious, I let this be as it is, for now. But I imagine future changes at this code location.

link_pdf()

At last, after the PDF has been written to the disk, the article HTML needs a link to the PDF in the article header.

display.py - link_pdf()

def link_pdf(soup, urn):  """  Adds the link to the PDF into the article header, represented by a  pdfimage.  """   pdfimage = "3cd97bab8bb20288768b35fd72979ec3bbf4b2a8.png"   header = soup.find("header")  divs = header.find_all("div")  if len(divs) < 1:  print("Call article_pdf() last before printing html")  return   div = divs[1]   figure = soup.new_tag("figure")  div.insert(1, figure)  anchor = soup.new_tag("a")  figure.append(anchor)   anchor.attrs.update({"accesskey": "p",  # "download": "",  "href": f"./pdf/{urn}.pdf",  "target": "_blank",  "type": "application/pdf"  })   # Inject the PDF Icon  img = soup.new_tag("img")  anchor.append(img)  img.attrs.update({"src": f"./idee/website/image/{pdfimage}"})

This is really the last thing done before the article HTML is written to the disk and shown in the Firefox web-browser.

If everything is copy-edited and all page-breaks are nicely set for the PDF, then the next to do is the handover to the deployment via git.

IdeePublish

The handover to the deployment is done with the command IdeePublish .

publish.py - imports and debugging

#!/user/bin/python3 """ Publish generated HTML and attached assets to the idee website git.  @date: 2026-02-05 """  import sys import os import string import shutil import getopt from pathlib import Path from bs4 import BeautifulSoup from bs4.builder._htmlparser import HTMLParserTreeBuilder  ...  if __name__ == "__main__":   MARKDOWNFILE = None   try:  opts, args = getopt.getopt(sys.argv[1:], ["o"])   except getopt.GetoptError:  print("No Parameter given")  sys.exit(2)   if len(args) == 0:  print("No Parameter given")  sys.exit(2)   MARKDOWNFILE = args[0]  if not MARKDOWNFILE:  print("No Parameter given")  sys.exit(2)   publish(Path(MARKDOWNFILE))   sys.exit(0)

The code shows the imports and the __main__ part used to call the publish function from terminal for e.g. debug purposes. The ellipsis shown in the middle of this code snipped stand for all the functions in between, which in the following will be shown one by one.

Function publish()

The publish function has not very much to do. It boils down to the following steps:

• adjust resource links to fit the target location
• move the assets to the target location
• copy the markdown file to the author directory of the idee website project.

The last step is important to have the source of the published article available for corrections, which might be required later.

Often something is found to correct directly after the publishing. Copy-editing sometimes fails, especially if the author is also the copy-editor, as in my case. I therefore decided not to cleanup the directory where authoring took place, but to delete that manually some days after publishing.

publish.py - publish()

def publish(filepath: string):  """  Creates an html web page frm the markdown file.   The html web page is fully flavored for that site, including the optional  linked in pdf and audio versions and content licence information.   Returns  -------  None.  """  mdpath = Path(filepath)  workdirpath = mdpath.parents[0]   # check that this a designated workdirectory  idee = workdirpath / "idee"  if not idee.exists():  print("Use first IdeeFolders to designate this location as"  " workdirectory, and use IdeeDisplay to generate idee-website"  " styled HTML"  )  sys.exit(0)   htmlpath = workdirpath / mdpath.stem  htmlpath = htmlpath.with_suffix(".html")  # read html  if htmlpath.exists():  builder = HTMLParserTreeBuilder()  with open(htmlpath, 'r', encoding='utf-8') as pf:  published_html = pf.read()  pf.close()  else:  print("Use IdeeDisplay to generate idee-website style HTML first")  sys.exit(0)   soup = BeautifulSoup(published_html, builder=builder)  resource_paths(soup)  html_doc = soup.prettify()   outpath = workdirpath / "idee" / "website" / "article" / mdpath.stem  outpath = outpath.with_suffix(".html")  with open(outpath, "w", encoding='utf-8') as outfile:  print(html_doc, file=outfile)  outfile.flush()  outfile.close()   copy_assets(workdirpath)  # copy the markdown file itself also  shutil.copy(mdpath, idee / "author")   print(f"Published to: {outpath}")

Function resource_paths()

Adjusts the relative paths to the articles assets as they are required by the web-sites directory structure.

publish.py - resource_paths()

def resource_paths(soup):  """  Change the resource paths to fit for the idee website  """  for i in ["src", "href"]:  tags = soup.find_all(attrs={i: True})  for tag in tags:  p = tag[i]  p = p.replace("./idee/website/", "./")  p = p.replace("./", "../")  tag.attrs.update({i: p})

Function copy_assets()

The function copy_assets() copies the assets in the authoring subfolders to their target location in the web-site folder structure.

publish.py - copy_assets()

def copy_assets(workpath):  """  Copy the assets located in subfolders to the respective subfolder in the  idee-website  """  for d in ["audio", "files", "image", "js", "pdf", "qrcode"]:  wd = workpath / d  wtd = workpath / "idee" / "website" / d  for f in os.listdir(wd):  wdf = wd / f  if os.path.isfile(wdf):  shutil.copy(wdf, wtd)

Changes in vimrc

There is just one more thing to mention. My vimrc file, which by default is in the folder ~/.vim/ , got some additional lines defining a key mapping in normal- as well as in insert-mode to run the command IdeeMeta .

" Markdown autocmd for idee ftplugin augroup md  autocmd!  autocmd FileType markdown imap <C-y> <C-O>:IdeeMeta<CR>  autocmd FileType markdown nmap <C-y> :IdeeMeta<CR> augroup END

This makes it very smooth to insert the comments used for meta data and to control aspects of the HTML/PDF creation.

Deployment

Deployment is a separate topic. I just teaser here, that I commit everything placed into the idee web-site project into git. During this commit RSS-feed, an archive page for the month, the index page of the web-site and the sitemap.xml are updated.

Those need then a second commit. It seems not possible to prevent that.

After pushing to the server-git, a client-git on the web-server is triggered automatically to fetch the latest changes, by which these become life in the web-site.

Two commits and one push to set a new or updated article life.

That's the topic of a future article, stay tuned.

Final Notes

Obviously this code is highly specialized for my own purposes. It is not meant to be copied, pasted and used as is in any other setting.

It shows, how I adjusted my own workspace to my own needs, which tools I used for this and how I utilized them.

I'm pretty sure it can be adjusted to serve the needs of someone else, and it could probably also be developed further to become customizable. At the present time I have no plans to do this.

However, I hope this to be a showcase, how to get rid of all the 3rd-party scripts loaded from 3rd-party servers doing not much more than exposing web-site visitors to private data sellers.

If you think it worth to derive a customizable version of this, usable by a broader audience, go ahead and do it. Its creative commons zero after all.

Or just ask. If someone asks for it, I have an incentive to go on with it.

Footnotes

1. The Cosmological Constant as Event Horizon ; Enrique Gaztanaga; Symmetry, volume 14; Multidisciplinary Digital Publishing Institute; DOI: https://doi.org/10.3390/sym14020300 ; 2022-02-01

2. XClacksOverhead.org ; XClacksOverhead.org; X-Clacks-Overhead

Taking England's all cause mortality data literally

2021-12-18

Frank Siebert

Update 2022-09-17: After migration from wordpress one picture got missing, it is now linked correctly. I also realised that I mentioned people dropping out of the the vaccinated group without getting registered in the next group. The word vaccinated is now corrected into unvaccinated .

The Data from the ONS dataset "Deaths by vaccination status, England" ^{( 1 )} from the "Office for National Statistics", got big attention. Information like this is pretty much unavailable everywhere else. In Germany I do not know of any source providing such information.

I already referred in an Article to the work done on this data set by Norman Fenton and Martin Neil" ^{( 2 )} .

Follow me into a deep dive into some aspects of the data, and what it tells us if taken literally.

Conclusions

The Data about all cause deaths by vaccination status England, from the Office of National Statistics, makes the following interesting statements:

• The vaccinations increase the all cause death rate of the unvaccinated.
• It is possible to get vaccinated with both doses on the same day

This, I want to emphasize, is not a statement from me. It is a statement encoded in the data from the Office of National Statistics. I explicitly emphasize, that I do not believe in the truth of these two statements.

In the following part you may follow my investigation into the data, to dig out these two statements together with me. As I dug even deeper, I came to a different conclusion. Feel free to make your own.

Investigating the Data

On Tab 4 of the Excel-File provided, all cause mortality data is provided with the following column names: "Week ending", "Week number", "Vaccination status", "Age group", "Number of deaths", "Population", "Death rate per 100,000", "Lower confidence limit", "Upper confidence limit".

By chance you might miss that one column, which comes without any column-header, featuring some 'u' entries to mark these lines as "unreliable". The criteria for this flag would be probably of interest, but it is mot documented below the table. From the position of that column we can tell, that it refers to the "Death rate per 100,000", as do the two confidence limits as well.

But we learn from documentation below the table: Figures are based on provisional mortality data and the Public Health Data Asset (PHDA), a linked dataset of people resident in England who could be linked to the 2011 Census and GP Patient Register.

The table provides the data separated into 4 age groups and 4 vaccination statuses, the age groups '10-59', '60-69', '70-79 and '80+_, the vaccination statuses 'Unvaccinated', 'Within 21 days of the first dose', '21 days or more after first dose' and 'Second dose'.

Why ONS decided to use only one big age group for the 10 to 59 age old people stays a mystery for me. It would have made the life of an uncounted number of statisticians much simpler, if the age groups would have been aligned with those of the vaccination status tracking.

Calculated age specific death rate versus provided age specific death rate

For counting numbers of deaths by vaccination status and age group and calculating the death rate by dividing deaths through the group size, you wouldn't need a confidence interval, if you knew those numbers.

But if you not know these numbers by counting, or do not know them exactly, wouldn't you name a confidence interval for each number you are not sure about? Can we find out more about the source of the confidence limits and "unreliable" marker?

Let us take some lines to compare the self-calculated death rate with the provided death rate. On June 4th, the unvaccinated age-group 10 to 59 registers 125 deaths and a population size of 11,832,842 people.

To calculate the death-rate per 100,000 we just multiply 125 with 100,000 and divide this trough 11,832,842.

125*100,000/11,832,842 = 1.056381

Showing just one digit behind the decimal comma, our result does match the result in the sheet. But we should take a line where more digits are available, it does not matter whether these are in front of or behind the decimal comma, to get a better resolution in our comparison.

The June 4 unvaccinated 80+ entry seems to be suitable, not being marked as unreliable by a lower-case "u".

209*100,000/75,045 = 278.499566

We get a perfect match of self calculated and provided death rate also for this line. This is the point, where it is probably better to start using the spreadsheet we are sitting in front of, to make that comparison for all lines in search for a difference.

Doing this, you can discover that all self-calculated death rates match with those provided as age-specific rate per 100.000. In a number of lines the death rate is omitted by ONS, in some other lines a lower-case "u" indicates unreliability caused by too low numbers.

This u-signal raises questions by itself, it might or might not be related to the confidence interval, but I didn't see such information in the documentation till now. What we found out by the investigation: The age-specific rate per 100.000 seemingly was not altered by any additional information. It is really just calculated by most simple math.

A look at the group transitions

What happens at time of vaccinations in regard to the death toll?

To look at this, we first use the death rate of the two neighboring groups, and visualize less prominently the population change, which does indicate the vaccine uptake, while at a lesser rate it also includes the population change by death. But as that second aspect is not dominating, we ignore this for a start.

We start our view with the age class 70 to 79, because the age class 80+ did start vaccination in 2020 and would not provide the fill view on the scenario. We do not start with the group 10 to 59, because that group is much to big and vaccination uptake of its undisclosed subgroups is spread over a long time period.

The age group 70 to 79, being a small age group embedded between two other small age groups is the perfect choice to have a start.

Graph 1

This graphic explains us, that the death rate of the vaccinated follows in very good approximation the group-size changes of the "Within 21 days of the first dose"-group.

If this is puzzling you, then you are not alone.

If we take this result literally, we have to claim that the first dose of vaccine kills the unvaccinated.

The wild peaks up and down in the death rate of the "Within 21 days of the first dose"-group starting around CW 17 are due to the small population size, and must not bother you at all.

If we do not take this result literally, then we need to think about the question: What is wrong with this data?

Three probable explanations for such an effect where given by Norman Fenton, Professor for Risk Information Management, and Martin Neil, Professor for Computer Science and Statistic, both teaching at the Queen Mary University at London, asking: "Is vaccine efficacy a statistical illusion?" ^{( 3 )} :

1. A delay in the reporting of deaths could be responsible, leading in a shrinking population to a spike of the death rate, which follows inversely the rate of shrinking. Officially this explanation cannot apply, since ONS claims the data is based on the day of deaths, not on the day of death-reporting.
2. The population of the unvaccinated probably got underestimated, which would lead to a growing death rate as the population shrinks.
3. Deaths are probably wrongly categorized, meaning vaccinated deaths are attributed to the unvaccinated group.

In the end also other explanations or a combination of explanations might be possible. We don't yet know the explanation, but we know that something is wrong with this data, because we have to believe that unvaccinated are not dying of the vaccination of others. So the data as we see it requires an explanation. To claim its true, that unvaccinated people are dying because others get their vaccine, is an insult on our common sense.

We found an artifact in this data, which shows definitely an effect not existing in real life. As we do not know the cause of that artifact, it seems impossible to remove it reliably. But probably it is, lets take a deeper look.

Death rates only by age group

For further analysis, I installed PostgreSQL ^{( 4 )} and wrote some SQL views. This way its best documented, what the respective view does exactly to produce the graph shown here ^{( 5 )} .

Graph 2

This graph shows us, that nothing special happens with the overall death rates. The highest death rate in all age groups is in CW 3, pointing out that this is not something vaccination related, but simply a seasonal increase as usual in the winter season.

Neither the COVID deaths nor the Vaccination deaths are expected to show up here dominantly and the all cause mortality was in decline where the strange increases in all cause mortality in unvaccinated shows up in the ONS data.

This can be seen as additional indicator, that the strange late death-spike in the unvaccinated group is just a statistic-only-effect without real-life relevance.

Death rate inoculated version unincolated

I'll focus in the further deep dive mainly on the age group 70-79. Less as deep dive but just to have it, I created also a simplified view ^{( 6 )} distinguishing only two vaccination status groups, the group of inoculated people which got at least one shot, and the group of uninoculated people which never got any COVID-shot.

Graph 3

A discussion of the conditions, which could explain such a graph, where the vaccination seems to cause deaths in unvaccinated, was provided by Norman Fenton, Professor for Risc Information Management, and Martin Neil, Professor for Computer Sciences and Statistics, both at the Queen Mary University in London, in their article "Is vaccine efficacy a statistical illusion?" ^{( 7 )} .

My intention is to go further with the investigation.

Checking the population balance

The data from Office for National Statistics is limited to a population, which took part in a Census in 2010 or probably one year next that, plus one additional criteria I forgot about. The point is: it is a closed population, which cam ne left only by death and which is not joined by anyone anytime in 2021.

Because it is a closed group of the population, we can check the changes in the sub-populations with simple math.

E.g. the death rates calculated by ONS use the population number of the week and the deaths of the week to calculate the death rate. Checking some weeks you can easily control that statement. This means, that the population given for the week is always the population at the begin of that week.

If we subtract the deaths of the week from the population of the week, we get the population at the end of the week, which would be the expected population at the begin of the next week.

If we would not have any age groups, this would already be all. But since we have age groups, one can move from one age group to another by having a birthday. But for a start we can ignore this to come back to this point later again. The separation into four different groups by vaccination status is also too complex for a start.

Therefore the next graph shows just for each age group, how much the previous weeks end population differed from the weeks start population. A negative balance (below zero) means, that people join the group by birthday. A positive balance (above zero) means, that people leave the group by birthday. At least that's the only known reason to change the age-group, as deaths take no other part in this balance check than in the calculation of the population at the end of the week ^{( 8 )} .

Graph 4

Looks strange? What's going on the age group 10-59 in week 13? An post from Anonymous from December 1st. explains, that the Census was closed on the 27th of March in 2011 ^{( 9 )}

Therefore my knowledge was weak, until the 27th of March in 2021 new persons (childs) still joined the age group 10-59 by birthday. Now I know better. For the age group 70-79, which I intended to focus on, is anyhow unaffected by this detail.

But it is definitely good to know that I do not have to bother too much because of missing people suddenly appearing where there is no possibility to do so.

We see in this graph that a lot of people leave weekly the group 10-59 to join the group 60-69. In the same time some less move on into the group 70-79, causing a negative balance. The group 80+ cannot be left other than by death, therefore it needs to have a negative balance.

Probably I should switch signs and let a positive balance make grow the group. I could easily do this, as it is just by definition the way it is now. But I stay with the current definition and implementation. It doesn't tell us anyhow, whether the group grows, since deaths are unaccounted for in the balance.

But we can also look at the population balance between the unvaccinated population and the "Within 21 days of the first dose" subgroups in the age group 70-79.

Graph 5

We see until week 4 a very good match, but then the two lines get out of sync. But this is generally correct, since everyone has to leave the second group after the 21 days.

Calculating the balance of the 21 days group

Just by look at the balance shown above, we cannot really tell, whether the 21 days balance fits with the unvaccinated balance. But since the two balances are tightly related, the 21 days balance can be easily calculated from the unvaccinated balance ^{( 10 )}

Graph 6

The dotted red line is the calculated balance, based on the transition of people from the unvaccinated group to the 21 days group. The orange line is the line given by the difference between end of last weeks population and begin of new week population as provided by the ONS Data.

The two curves are a very good match, with seemingly tiny mismatches between the 12th and 21st week. Taken literally, a big bunch of unvacinated people where either vaccinated at that time, all dying and not being able to join the 21 days group, or an extraordinary big of people had birthday in these weeks, making the unvaccinated disappear into the next age group.

The event peaks in week 16 and if we look into Graph 4 we do not see any special birthday event with change of age group reflected there for the age group 70-79. And death is also not an alternative, because if their death would have been registered, the peak in the graph would not exist.

We see people dropping out of the unvaccinated group without getting registered in the next group and without being registered as death.

The ledger not in balance. The books are not well kept.

But its not much, isn't it?

Zooming into the deviation in the group transition from vaccinated into the 21 days group

In Graph 4 we see the Net movement between the groups, not directly, but we get an approximation.

• About 12,000 move every week from age group 10-59 into 60-69
• About 9,000 move every week from age group 60-69 into 70-79
• About 5,500 move every week from age group 70-79 into 80+

Sometimes its more or less, but in the time frame we look at its a quite stable situation.

Looking at age group 70-79 we could also say - its a 10 year group, each year having 365 days, every day being as likely a birthday as any other. We can calculate the weekly number of birthdays shifting members above the 79 years as: New ₈₀₊ = Population _70-79 * 7 days / 10 years / 365 days/year ^{( 11 )} .

This calculation gives an upper limit for our expectation, in which range a population change of the group can possibly be during one week, without being very suspect. It is a very generous threshold, because:

• It is expected that most vulnerable older people of the group got vaccinated earlier, making birthday events with move to the age-group 80+ in the unvaccinated sub-group less likely than in the overall population.
• The unvaccinated people joining from the next younger age group are not included in this view, and those would narrow down the threshold even more.

At the other hand:

• Including the vaccinated people of this age group leaving into the age group 80+ would increase the threshold, but lets see if these about 5500 weekly age group changing birthdays in this age group would make a difference. If we get that impression, we can extend our SQL statement.

We can apply this view also to subgroups of that age group, which changes its size because of vaccination events.

In the following picture the dotted lines show the upper and lower threshold for the maximal expected birthday related deviation in the weekly population balance based on the size of the unvaccinated group of the age group 70-79.

Graph 7

We see a quite strong signal here, way outside the expected birthday related effects. '''In CW 16 the deviation of the population imbalance exceeds the expected birthday related effects by about 80,000 people. '''

What is the exact meaning of this signal?

About 120.000 People left the unvaccinated age group during the weeks 14, 15 and 16 without becoming member of the 21 days group. A possible move into the age group 80+ can not explain this. Looking at Graph 4 we see very clearly, that these 80.000 did not move to the 80+ group during those 3 weeks.

In Graph 3 the group of inoculated people shows a significant increase in week 16 running synchronous with a decrease in the uninoculated people at the same time. This probably means that the major part of these 120,000 people moved into another group. The question remains, in which group did they move?

Searching for the transition target group of 120,000 people

Since the move into age group 80+ definitely can not give any account for these 120,000 people, we have to take a look at other groups inside of the age group 70-79.

Graph 8

Now that our sight is schooled by the data analysis we did until this point, we can see immediately that at least the major part of these 120,000 unvaccinated people moved directly to the 2nd dose group. Do you see it?

About 120,000 people moved directly from the unvaccintated into the 2nd dose group in the calendar weeks 14, 15 and 16.

You may not ask me, how this is possible. I formulate only a statement made by the "all cause deaths data set" from the Office of National Statistics. It is not my own statement, it is a statement made by the data itself.

One probable reason comes to mind: Maybe they where vaccinated with Janssen / Johnson&Johnson single shot vaccine?

BBC reports at the 28th of May: "Janssen single-dose Covid vaccine approved by UK" ^{( 12 )}

This is the end of calendar week 21, which is much later. The single dose vaccination can not provide any explanation.

Aftermath

75,884 all cause deaths where reported in the age group 70-79 in the time frame covered by the ONS data.

If we do not believe, that the vaccinations are responsible for deaths in the unvaccinated group, and if we do not believe that some people get both doses on the same day, which significantly deviates from the official vaccination plan, what is the final conclusion?

About 120,000 unvaccinated suddenly being vaccinated with the second dose, while the complete all cause deaths where less in the same age group.

I can only draw the conclusion, that the complete number of all cause deaths can be covered by the 120,000 people which suddenly jump across two intermediate groups.

And these, I have also to conclude, where the surviving so-called "unvaccinated". I have to name them now "so-called unvaccinated", because they obviously where vaccinated and survived vaccination until the second dose was applied.

No one can tell how many non-surviving so-called "unvaccinted" died before getting the second dose, increasing the officiak death-toll of the unvaccinated group.

Going back to the phenomenon that the death rate of the unvaccinated seemingly follows for its best part the population size of the "Within 21 days" Group, as it is shown in Graph 1. Now that we see about 120,000 "unvaccinated" getting their 2nd dose in the weeks 14, 15 and 16, it is much more than just a guess, that this death rate of the "unvaccinated" is dominated by vaccinated people, which where wrongly accounted as unvaccinated.

120,000+ people aged 70 to 79 did not move into the 'Within 21 days of the first dose' group after getting their first dose.

This tells us also: Every single death rate provided for the "Unvaccinated" aged 70 to 79 in the ONS data deserves to be flagged as 'unreliable'.

Indeed, obviously the same has to be true for all other death rates in the age group 70 to 79.

The Office for National Statics did not deliver on its promise.

Give me a different satisfying conclusion, please.

Going Further?

Since the theme of this statistic, together with the choice for a well-defined closed group, established by itself a kind of double bookkeeping in the data, it was possible to identify a discrepancy in the data proving the bookkeeping as faulty.

Thanks to the nature of a double bookkeeping data set, it might still be possible to correct errors in this bookkeeping to some extend. Probably, if this turns out to work, I'll be able to provide additional insights in a later article.

Cognition is always temporary and always of individual nature. You decide for yourself whether you adapt the findings of others as opinions or whether you acquire own insights. My references are intended to help you with the latter, but you should always use other sources as well.

Do not believe, not even me, but examine and conclude for yourself.

Footnotes

1. Deaths by vaccination status, England ; www.ons.gov.uk; Release Date 2021-11-01
2. Professoren stellen die Frage: Ist Impfstoffwirksamkeit eine statistische Illusion? ; Frank Siebert; idee.frank-siebert.de; 2021-12-02
3. Is vaccine efficacy a statistical illusion? ; Norman Fenton, Martin Neil; probabilityandlaw.blogspot.com; 2021-11-14
4. PostgreSQL Installation ; wiki.frank-siebert,de/inst
5. View - ons.v_aggr_pop_deaths ;wiki.frank-siebert,de/inst
6. View - ons.v population and deaths by inoculation status and age group ; wiki.frank-siebert,de/inst
7. Is vaccine efficacy a statistical illusion? ; Norman Fenton, Martin Neil; probabilityandlaw.blogspot.com; 2021-11-14
8. View - ons.v_weekly_population_balance ; wiki.frank-siebert,de/inst
9. The impact of misclassifying deaths in evaluating vaccine safety: the same statistical illusion ; Norman Fenton, Martin Neil; Comment by Anonymous, 2021-12-01
10. View - ons.v_population_transition_21d ; wiki.frank-siebert,de/inst
11. View - ons.v population transition 21d deviation ; wiki.frank-siebert,de/inst
12. Janssen single-dose Covid vaccine approved by UK ; Philippa Roxby; BBC; via archive.org, 2021-05-28

Replacing WordPress

2022-02-16

Frank Siebert

Is it a story, or is it a technical documentation? Probably it is both, I only know it started with a specification of some kind and went on to become a solution. And now I try to compile the specification and the implementation notes into a journey description. My journey to Python and my new web representation and what I had learn on my way.

The code in this article has still hot needle quality.

Motivation

Part One

I think WordPress is a good platform to create a web presence. It is just a pain in the ass if you start to care about privacy. You start to remove all the included tracker features contacting google analytics, you continue with the removal of "social" media features, because you are not sure whether these communicate the respecitve platforms without pressing the social media button and you remove all those resource links loading resources from foreign servers, because the respective resource servers might be able to identify you and also the page you visited.

After this isolation of your web presence, you are halfway sure that the privacy of your visitors is save. Then you might receive a notification, that you should update WordPress or one of the plugins. ...

If you do not update, you might run into the situation where your web presence might become insecure for your visitors. If you update, you have to review everything again. Is your server isolation still in effect, or did the update reinstate some of the removed side-communications?

Part Two

I decided to learn Python and needed a project for this. I stumbled upon the article "Gitblog - the software that powers my blog" ^{( 1 )} , and I liked the idea to publish via

$ git push

I'd guess most developers are in agreement with this. Posting articles just the same way as you push new code to a git server - developers a bound to like this.

The Gitblock solution is based on Java and nothing is wrong with this. I used Java in software development for more than 20 years, with a noticable break where I mainly used ABAP to come back to Java again.

But since I wanted to learn Python anyhow, this was the ideal project to get a start with that language.

Project Duration

From start to go live it took slighly less than 2 month. The first Wiki entry dates back to December 31 ^st 2021, but this was nothing more than a note to myself. January 16 ^th the real drafting on the specification started and rarely anything from that first specification survived. But the most important requirements stayed stable and are met with the solution.

• No JavaScript
  • There is one article containing JavaScript, because the article contains a quiz. But articles are content, not publishing solution.
• All plain HTML+CSS static content
• State of the art semantic HTML
• Search (the exception from the static content)
  • Currently YaCy integration
  • Planned to have a python based Search

The website with all previously published articles migrated did go live on March 14 ^th . And migration was a really heavy topic. Most articles where originally written in one of my Wikis and then published in WordPress, but some of the very short video, audio or article recommendations where not written in the Wiki. Some articles got corrections maintained directly in WordPress instead of a correction in the Wiki with republishing afterwards. For some of those corrections I just made a comment in WordPress to make readers aware of the mistake.

In short: Inconsistency in previous publishing's made migration a major effort.

Other inconsistencies where:

• Articles maintained in Wiki had partly chapter headlines starting with header level 2 and partly starting with header level 3.
• Quotations where only preceeded with "Zitat:" and concluded with "Zitat Ende", but not everywhere, since I started this only when I started with the audio recordings of my articles - to make sure I do not forget to mention the start and the end of the quotation in the recording.
• Quotation where not enclosed in < blockquote > tags.

To provide state of the art semantical HTML I had to copy-edit every article during the migration. The good news is: The new setup will drive and support me to publish in a more constent manner.

Considering this major migration effort, I'm pretty proud the project did take "only" 2 month, especially since it was the Python learning project.

As it happens often on the way, I learned much more than just Python. I learned new things about PDF generation, fonts, git, regex, HTML, CSS, vim, the IDE spyder, the web server nginx and even more.

Requirement Specification

The requirement specification was subject to changes. As it often happens, this was mainly because it not only described requirements, but made also already assumptions about technical details of the solution.

A funny fact: I spend years explaining my own customers that it is important not to write requirement specifications with a technical solution in mind. The requirement specification should focus strictly on non technical scenario descriptions. The rationale behind this: Very often a customer would ask to eliminate work efforts caused by previously implemented workarounds. This workaround is viewed as tool by the customer, and following the customers suggestion leads to the implementation of yet another workaround. Very often you get a much better overall solution, if you sunset also existing workarounds, which is difficult, because those where so helpful in the past.

My previous publishing scenario

I use one of my MediaWiki siblings to collect information and I use also this wiki to create articles based on that information. This part of the publishing szenario stays in place. I considered to change this as well and to write articles in future in the editor vim, but I decided to keep infomation collection and article compilation together in one place.

To get this article published, together with its audio recording. I used the HTML export option of an PDF export extension.

I then used the editor vim with 3 regex statements to strip the header and the footer from that export, and, if necessary, also the references to categories, which would otherwise establish links pointing into the void when displayed in WordPress.

The remaining HTML was then pasted into one HTML input field in the Create Post UI of WordPress. Thus the page internal links in the table of contents and to and from the reference section of the page stayed functional.

If pictures were included, I uploaded these first to WordPress and used these uploaded pictures already in the wiki. That way the links to the pictures stayed as they were in the later WordPress version of the article.

All in all not too cumbersome a process, but with room for improvement. Especially when corrections where required it was much to easy to apply the correction directly in WordPress instead of doing the correction in the Wiki and to republish it. And this leads to problems on the long run. For some time I thought about some automation of text deployment to WordPress to mitigate this. But those thought are now obviously obsolete.

How do I want to do it in future ?

This chapter is from my early specification notes. I tried to figure out what I really want.

This is not really easy to tell. I'm still struggling to have one opinion with myself about this topic.

I'd like to edit my pages with MediaWiki markup or, as alternative, with markdown. From the implementation side it would be simplest to keep the editing process as I do it today and only to change the publishing.

The publishing and the result as shown in Gitblog is quite to my taste. However, this solution is based on Java, and I think 20+ years of Java is enough. I'd like to base my own solution on Python. Not because it is so much better than Java, which might or might not be the case, but because I decided to learn Python down to its depths and such a project is a perfect opportunity.

This does not mean, that I need to write everything from scratch, there are already a lot of modules in existence to build upon.

At the other hand I'd like to be able to write my articles also completely offline, just using vim as markup editor. But is this a realistic scenario? Am I not researching every detail anyhow online during the authoring? So many things you did read about and you are quite sure about, but you need a source as reference when you write it into an article. Will I ever really do authoring offline?

But why not both options?

During commit a pre-commit handler can check the mime type and do one thing if it is an html-fragment to be placed into an empty html page template, and do another thing if it is an markdown file with the extension md.

Looking at GitLab Docs ^{( 2 )} it is undeniable that powerful versions of markdown exist. However, installing GitLab means also to install a big bunch of software, which is not really smaller than wordpress. In search for a small minimalistic solution GitLab is probably out of scope.

To be honest, I'm not sure I ended up with less installations than Gitlab. But as you see in this text, I initially expected that would need to meddle with the HTML from the MediaWiki, as I did before. Fortunately specifications are a moving target, something we developers will often complain about.

How shall it look like in the Future

''This chapter is from my early specification notes as well. It was less off the mark than the previous chapter.

For a start it should look like before, just without those things no longer required. E.g. a logon is no longer required, since I push and merge new articles to the server, instead of logging in and using an authoring front-end.

It looks different, but not too much different

Search

Initially the search will use my YaCy instance. I have to look how well this integrates.

Yes, YaCy is integrated. But I consider the current search integration as improvable.

Site Pane?

Is a side pane required any longer? Probably not, I'm not sure.

No site pane any more.

Header Collapse or not?

Can I collapse the header with the site navigation during scroll down and make it available when the user starts to scroll up? I mean without JavaScript, only with CSS? I will see.

This point got no priority at all. Nice idea probably, but in the end I didn't care.

Small Header

The header will become smaller, since I will shorten the main site name to "Idee" with a smaller "der eigenen Erkenntnis" and I will write it in small-caps to hide the problem with uppercase "Erkenntnis" not fitting exactly to the lowercase "e" at the end of "Idee". And the Header-Text will move on Top of the Header Picture. ''Forget it. Ok, the header got smaller, but that's it.

Article PDF

Every article will get an PDF-Download Button. The PDF is not necessarily optimized for print and offline reading, but it is nonetheless a good idea to simplify the access to references in the reference section via QR-Code for the respective links.

Article PDF is implemented and also a possibility to suppress its generation for low value content, e.g. if the "article" is just a recommendation note.

References in the PDF are rendered as in the online version, but showing additionally the HTTP address as text. QRCode creation for every reference does not take place.

Article Archive

WordPress shows am archive drop down. That needs scripting and dynamic population. An alternative would be the generation of one archive page, which allows drill down to the year, which allows drill down to the month. Such pages can stay unchanged, as long as I do not change the portal part. as soon as the respective month or respective year passed by. The usability is most probably not less than a drop down, which at some point gets a bit messy to scroll on small screens.

Implemented as described. Only a portal change does in the chosen implementation not require a regeneration of pages, which is a huge improvement compared with the initial specification.

Sizing Pictures?

Should I size pictures during commit? Should I sample audio files to a number of different qualities? A lot of options are open now, with the development of an own page factory.

The question-mark in the title can be answered Today with No.

Picture based article selection

I could create a picture gallery to select articles by picture. But then probably I should create a picture for every article... . Not really, I also, in rare cases, do not create audio. I wouldn't force myself into picture creation, where the picture does not add value.

The original text does hint it already. Nothing in this regard has happened.

Semantic Web

Articles will have state of the art HTML5 article structure. This needs some intelligent logic if it comes to the correct use of tags like the cite-tag. I probably need to think about the markdown and MediaWiki representation of the HTML cite-tag to make this one working nicely.

I obviously meant quotations. The markup representation for quotations is the respective HTML tag < blockquote >. The mentioned < cite > tag could probably come into use in my articles as well in the references, but this is probably not a could idea. Possible that I'll introduce this later.

However, a lot of semantic is simple. The article content resides in the article-tag. The article-tag contains a header-tag, whose headline and media are descriptive for the complete article, like the QR Code of the URL, the PDF file and the audio file. Video is not planned.

The HTML head meta-tags for articles and the og meta-tags bring a lot of invisible semantic to the page.

A lot of options. In the end I will strip this text down to those things, which made it into the product.

It's all in, apart of the cite tag, which was an error in the specification.

Citation

That would be an interactive page function.

• Reuse Citations I made in various citation formats.
  • Click at a function link at the footnote. the citation gets shown and a citation format can be selected.
  • The result can be used via copy-paste buffer.
• Cite statements made by me in various citation formats.
  • Mark a text passage in my article and a citation function link gets shown.
  • Then as above.

Yes, this function needs java-script, which would be a draw back from the plain HTML philosophy I started with. Citing myself a lot of other publications and knowing about the effort to create the citations as I need them, I think such a function is worth to be scripted.

Plain HTML is not a religion. It is rather: Avoid scripting where it is not required. Ok, nice idea to make it simple for others to cite me. But it is not implemented and probably will never be.

JavaScript

JavaScript can be disabled in the browser without any impact for the casual user, Only "extended features" may rely on JavaScript, as the citation feature. Features not enabled due to disabled JavaScript stay invisible to the user.

I translate myself for myself :-). Function-links are written into the page via JavaScript. If JavaScript is not enabled, the page will not contain any malfunctioning function links.

Till now I did not need to follow this specification, since the implementation doesn't use JavaScript anywhere. But the specification stays valid.

Sitemaps

The "Sitemaps XML format " ^{( 3 )} description explains the concept and the XML document structure of sitemaps.

RSS

Updates have a reason, most probably additional or corrected information went into the article. To make an announcement of such changes is imperative for an information provider, and the RSS.xml is the place for this.

Monthly Archive

As sitemaps can be structured by one or more sitemap index files, it does make sense to use this to structure the sitemaps by "yyyy-MM", getting one sitemap per month.

Thus its probably simple to create a monthly page of articles to be selected by the user in the archive overview created from the sitemaps index.

The first sentence describes how the implementation was done later. But the second sentence was too optimistic. The sitemap, if not pepped up with extensions, does not contain enough information to create monthly archive pages from it.

HTML5 Article (with prepared location for portal injection

This is the article HTML draft. it contains a line with div id="main", which I planned to be the place where I place the portal part via Phyton. That div turned out to be unnecessary, instead an xml comment is now placed after body and before main tag as include instruction. The include is performed by the nginx web-server.

<!DOCTYPE html> <html lang="de-DE" xml:lang="de-DE" xmlns="http://www.w3.org/1999/xhtml">  <head>  <meta charset="utf-8"/>  <meta content="pandoc, fs-commit-msg-hook 1.0" name="generator"/>  <meta content="width=device-width, initial-scale=1.0, user-scalable=yes"   name="viewport"/>  <meta content="2022-01-19T16:05:43" property="article:modified_time"/>  <meta content="2020-10-15 09:49:27" property="article:published_time"/>  <meta content="Frank Siebert" property="article:author"/>  <meta content="Idee" property="og:site_name"/>  <meta content="de-DE" property="og:locale"/>  <meta content="The Article Title" property="og:title"/>  <link href="../website/css/fs.css" rel="stylesheet"/>  <title>  The Article Title  </title>  <style>  <!-- styles by pandoc -->  code{white-space: pre-wrap;}  span.smallcaps{font-variant: small-caps;}  span.underline{text-decoration: underline;}  div.column{display: inline-block; vertical-align: top; width: 50%;}  div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}  ul.task-list{list-style: none;}  </style>  </head>  <body>  <div id="main"> <!-- Portal injection parent -->  <main>  <article>  <header>  <h1>  The Article Title  </h1>  <div>  <time datetime="yyyy-MM-dd hh:mm:ss" pubdate="true">  yyyy-MM-dd  </time>  <address>  Author Name  </address>  <!-- probably PDF download link location -->  </div>  <!-- probably audio player location -->  </header>  <!-- article content (paragraphs, toc, headlines (< h1), images, footnotes)  </article>  </main>  </div>  </body> </html>

Scenario

This is still specification before the development started. It seems to be repetition, but it is not, because it contains a decision not made before. But it contains also things which should not be written into a scenario. If you wear the hut of the customer, the architect and the developer all in one person, then there is no second person taking care to write the specification correctly.

Obviously I'm not completely sure about the scenario. But if it changes over time, then that's a common thing often seen also in other projects.

Decision: Material collection and writing happens in a MediaWiki. The export may happen with the current export tool, or it might happen with a Python based Wiki-page parser ^{( 4 )}

Aspiration: I want to have useful meta tags generated during the commit. The Author Information and the date of publishing, the date of update and a documentation of changes should be automatically processed into an HTML meta information and into a standardized representation in the visible text. This is important to ensure that corrections are processed in a transparent, reader friendly manner.

For the sake of usability, a commit should not lead to an automatic release of the text. The commit is for the draft version only. This basically means that I will work in branches and the final publishing is done with a merge into the master branch,

Forget this branching explanation. Yes, commits generate HTML for review, nut branches are not necessary and therefore also no merge. The final publishing is done via the git push command, as explained earlier.

During the merge into the master branch on the server:

• The HTML is processed to contain a header, a style-sheet, meta information and change markers if the document had not been merged for the first time.
• The page is fed into a search engine for indexing
• The page is fed into an rss feed generator to provide a new entry in the rss feed.
• The page is fed into a sitemap generator to provide an updated sitemap

In the end everything is done during commit, with the exception of search engine indexing, which can be done by the YaCy-Search Engine only after publishing.

Search Index

And even more specification, if you like to call it such. Probably it is more am investigation of options regarding search.

For Python some search index implementations exist. There is one Doit-Yourself-Example by Bart de Goede ^{( 5 )} , at the opposite end of the spectrum we find Gensim ^{( 6 )} , which probably can do much more than just index, and there is a module named Whoosh ^{( 7 )} , and there is rank-bm25 ^{( 8 )} , which implements multiple variants of the bm25 search algorithm.

I tend to base my search on the latter module, and I'm curious how well this will work.

Search Index Related Learning Material

• "Improvements to BM25 and Language Models Examined" ^{( 9 )}
• "What is the difference between Okapi bm25 and NMSLIB?" ^{( 10 )}

Implementation

With the chapter "Toolchain" the implementation started. The chapters are sorted by initial implementation sequence.

Toolchain

MediaWiki-Tools git ~/projects/wikitools/

This is the git used to implement the tools to access the MediaWiki instances. The default instance used is my private sammel-wiki, but there is no reason why I should not also access my installations-wiki to create postings from it. Well, the language probably, since my blog is in German language.

The language problem is solved with the creation of two sites, one in German and one in English

The wikitools project git existed already, hosting the code for a program "reference.py" to scrape Webpages for the creation of a reference tag stored in a new created reference wiki-page for the scraped website.

Related to WordPress-replacement-project is the new tool "export.py", which extracts a wiki-page with expanded templates as MediaWiki markup file. The output of this tool is placed into a configured directory, which is, how convenient, the authoring directory of the authoring git.

Authoring git ~/projects/idee

Authoring takes place in the folder ./author/ , for a start via MediaWiki files. This means I can also use my MediaWiki instances for authoring and afterwards use the export.py from my wikitools to save the article as "authoring source" into this folder.

During git commit the commit-msg hook implementation checks for committed ./author/*.mediawiki files to be processed.

TODO: Consider options to structure this in ./author/yyyy/MM/ folders.
DONE: The result is NO.

The respective mediawiki files are processed into plain HTML by the method pandocmw() , pandoc beeing the conversion tool used.

Processing results are stored in:

• ./plain/ - the plain html files
• ./website/image/ - the image files

The plain html files are further processed into PDF files by the method pandoc-html-pdf() , pandoc again beeing the conversion tool used.

Processing results are stored in:

• ./website/pdf/ - the pdf files

Created HTML and PDF files open automatically in Firefox for review.

At this point of the processing the Pictures as well as PDF files are supposed to be final, at least after some possible round trips of review and correction.

TODO: check the creation of an asset list for each authored article, to prevent the deployment of the article without the corresponding pictures and audios.
DONE: The result is NO. The risk of this to happen is minimal and it is also very fast corrected, if it should happen

TODO: one option to simplify the commit of all required assets is the creation of an asset list for the committed article. Probably this can be in the form of a prepared commit message for these assets.
DONE: The result is NO. All git comments can be issued in the root of the git repository, making sure everything is included.Just "git add .", "git commit", that is simple enough.

During git commit the commit-msg hook implementation checks for committed ./plain/*.html files to be processed.

The respective plain html pages are processed by the method injectportal() into webpages of a website via:

• the injection of the portal into the page
• the placement of the PDF accesss link, if a pdf with the same name exists
• the placement of the HTML5 audio player, if an audio with the same name exists

The processing results are stored in:

• ./website/article/ - webpages containing articles

Then the website is updated via:

• the update of the sitemap,xml
• the update of the feed.xml
• the update of the index.html featuring the latest post as first entry.

Processing results are stored in:

• ./website/ - Entry point of the web representation

./website/ contains all website related content.

Resulting in the structure:

 website  ├── article  ├── css  ├── media  └── pdf

Privacy Statement and such administrative overhead will be deployed as article and linked as special page in the portal.

TODO: Think further about URL compatibility with the current WordPress site.
Option: I have exported data from the mysql table, which enables the creation of a redirect list.
DONE: Compatibiliy is a must and it is ensured. It is important not only for the redirect, but also to preserve the correct dates of the articles. The stem of the article page serves as urn to access the article data in the publishing list.

This git is a client git, connected to the server git. Deployment to the server is done via git merge .

Autoring git - Configuration

The configuration of the git gets stored and versioned in the git repository. The path to the configuration is ./config/ . The implemented hooks are part of the configuration and are stored in ./config/hooks/ , the preexisting examples are stored in ./config/hooks/samples/ .

./config/gitconfig

#!/bin/bash # configure the wiki  # We develop hooks and want version control for that git config --local core.hooksPath ./config/hooks  # We want to easy reading of German äüö in the file names git config --local core.quotepath off  # We provide some variable override options in a modified template git config --local commit.template ./config/commit-message  # We process the files committed and need absolute file paths from $GIT_DIR # written into the commit-message git config --local status.relativePaths false

The configuration settings are applied with the above shown bash script.

Folder Structure

To give you complete overview of the final git folder structure, here it is:

frank @Asimov:~/projects/idee$ tree -d . ├── author ├── bash ├── config │ └── hooks │ └── samples ├── generator │ └── __pycache__ ├── nginx ├── plain ├── test └── website  ├── archive  ├── article  ├── audio  ├── css  ├── env  │ └── bootstrap  │ └── css  ├── files  ├── image  ├── js  ├── pdf  ├── portal  ├── qrcode  └── sitemap  25 directories

The folder website/env/bootstrap/css contains the CSS to format the YaCy search result page. In the current implementation I refrained from merging somehow the portal header into that page. Probably I could convince YaCy to return the results as XML and to render a portal page via XSLT, There is room for improvement.

The folder /generator contains the Python part of the project. If you consider software development and content development as two different projects, then the git contains the development project as a nested project.

To get a re-usable software product from this, it is required to separate the projects into separate git repositories. But for initial development the combined repository saved a lot of time.

Server git: /home/git/idee.git

This is the server git, and as such it is without work directory. When the pushed changes had been merged, a hook needs to take care to write the content into the web-server directory.

Not all content, but the content belonging the website as documented above, needs to be processed.

The implementation uses a simple fetch by a client git located in the web-server directory.

Export

There are tools you can install in your MediaWiki instance to support the export to PDF and to HTML. However, these require you to change the wiki installation and the result might not be tailored to your need,

To publish articles I decided to write my own export tool, extracting a single page containing the composed article, with all included templates expanded. Mediawiki ships with a special page Special:Export, which uses the same API function as used in my implementation. The API call was already implemented in the module mwclient.py, but to get it to function with long pages I had to change the respective GET request into a POST request. I informed the developers via their Github issue tracker with the message "expandtemplates should use "post" instead of "get" · Issue -272 · mwclient-mwclient" ^{( 11 )} .

The current implementation of the export.py is by no means beautified. I just began with python and I'm pretty much unaware of established coding conventions. As I progress with python, things will get nicer over time.

export.py

wikitools git repository

I let a lot of comments survive, which document also some of the wrong ideas I had. E.g. during the migration I used the Pandoc feature to download the images from their web location on my WordPress installation. Pandoc creates own filenames for the pictures via SHA1 hash during this process.

Afterwards I thought I should change the export function to enable Pandoc also to download images from the Wiki. But the wiki requires authentication and overall the identifier for the image might Image:, File: Bild:, Datei:, and in additional languages you might get additional alternatives. I'm not sure whether Pandoc does address all those correctly, but at least that's SEP ^{( 12 )} as long I do not meddle myself in that soup.

Now nothing is done in the export, the media download in Pandoc is deactivated and the image path is adjusted right after HTML creation at a point, where I meddled with that path anyhow already.

#!/usr/bin/python3 """ Export MediaWiki Pages with expanded templates and page includes.  @author: Frank Siebert @website: https://idee.frank-siebert.de @license: https://creativecommons.org/publicdomain/zero/1.0/deed.en @date: 2022-03-15  """  import sys import os import getopt import termios import fcntl import subprocess import time import re  from pathlib import Path import configparser from termcolor import colored from mwclient import Site from mwclient.errors import LoginError  HELPTEXT = 'Usage: export.py [-w \'wiki\'] \'Page_Name\'\n'\  '\n'\  '-w \'wiki\' Name the wiki to be used, using the section\n'\  ' name in the configuration file.\n'\  '\n'\  'Page_Name The page to be exported. In case of spaces either\n'\  ' surrounded by \' or with _ instead of spaces.\n'   def askforkeypress(prompt, keylist, onerror):  """  Ask and wait for user input.   Parameters  ----------  prompt: Str  The promt shown to the user to ask for input.   keylist: List  A list of characters as possible input keys.   onerror: Object  An object to return to the caller in case of an error.  """  fileno = sys.stdin.fileno()  oldterm = termios.tcgetattr(fileno)  newattr = termios.tcgetattr(fileno)  newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO  termios.tcsetattr(fileno, termios.TCSANOW, newattr)   oldflags = fcntl.fcntl(fileno, fcntl.F_GETFL)  fcntl.fcntl(fileno, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)   # stay in the same line to wait for input  print(prompt, end=' ', flush=True)  char = None  try:  while char not in keylist:  try:  char = sys.stdin.read(1)  time.sleep(.1)  except IOError:  char = onerror  except ValueError:  char = onerror  print(char)  finally:  termios.tcsetattr(fileno, termios.TCSAFLUSH, oldterm)  fcntl.fcntl(fileno, fcntl.F_SETFL, oldflags)  return char   def export():  """  Export the page as mediawiki markup.   Uses the API used by Special:Export including templates  https://wiki.frank-siebert.de/script-inst/index.php?title=Special:Export   For long pages the use of POST is important. I changed the library function  in this regard. A respective fix awaits its merge into the mwclient module.  Request Type: POST  Request Parameters:  catname=&pages=Replacing+Wordpress&curonly=1&templates=1&wpDownload=  1&wpEditToken=%2B%5C&title=Special%3AExport  """  print(pagename)  # Login  host = config[CFGSECTION]['Host']  scriptpath = config[CFGSECTION]['ScriptPath']  user = config[CFGSECTION]['User']  password = config[CFGSECTION]['Passwort']  exportdir = config[CFGSECTION]['ExportDirectory']  references = config[CFGSECTION]['References']   site = Site(host, path=scriptpath)  try:  site.login(user, password)  site.get_token("login")  except LoginError:  print("login failed")   for result in site.search(pagename, what='title'):  print(result)  page = site.pages.get(pagename)  if page:  print(page)  # expand = page.templates().count > 0  # might fail if client.py is updated, because  # def expandtemplates(self, text, title=None, generatexml=False)  # sould use post instead of get  wikitext = page.text(section=None,  expandtemplates=True,  cache=True,  slot='main')   # load page until it is no longer changeing  # stop waiting after 5 seconds  webpage = wikitext  time.sleep(.5)  i = 10  while len(webpage) != len(wikitext) and i > 0:  webpage = wikitext  time.sleep(.5)  i -= 1  # make clear, which one is not used from now on  webpage = None   # Later we might need a <reference/> tag to identify the  # location where footnotes shall be placed.  # Lets check non-existance of the tag and existance of the  # reference section.  # Insert the tag now, if it is not present at the expected place.  # In vim the regex :%s:^=.*Fußnoten.*=\n<references.+/> finds  # the german footnotes with the reference tag in the next line.   # strip down to headline text only  ref_cfg = references.strip().strip('=').strip()   # I tried repr(pattern) to get the raw string, but got it with  # ' at the start and end  # repr(pattern)[1:-1] would strip them off, but this is easier  # to read and to write   # requires re.M  r_ref_cfg_pattern = r"" + "\n=.*{}.*=.*".format(ref_cfg)  # a negative lookahead for the reference tag  r_nreference = r"" + "(?!\n.*<references.*/>.*$)"  # ok if footnote header found without reference tag  r_okpattern = r"" + r_ref_cfg_pattern + r_nreference   okcheck = re.compile(r_okpattern)  exists = okcheck.search(wikitext)  if exists:  print(colored('\nWARNING:', 'yellow'),  "<references/> tag is not in the expected location")  print("Automatic insertion of the tag takes place.")  wikitext = okcheck.sub(exists.group(0) +  "\n<references/>", wikitext)   # replace category references  categorypattern1 = re.compile(r"(.*)",  flags=re.MULTILINE)  wikitext = categorypattern1.sub(r"\1", wikitext)   # replace category references, take care not to touch Images  categorypattern2 = re.compile(r"\[\[[K|C]ategor.*\|(.*)\]\]",  flags=re.MULTILINE)  wikitext = categorypattern2.sub(r"\1", wikitext)   # A mediawiki extension enables me to embedd images from my  # idee demain into articles I write in the wiki just by pasting  # the URL in to an otherwise empty paragraph.  # https://something....filename.ext  # This, who wonders, is not recognized by pandoc.  # I need to beautify these image links.   # only if the line starts with http  # transitional code for the migration  imagepattern = re.compile(r"" + "^http(.*)png", re.MULTILINE)  wikitext = imagepattern.sub(r"[[Image:http\1png|No Caption]]",  wikitext)   imagepattern = re.compile(r"" + "^http(.*)jpg", re.MULTILINE)  wikitext = imagepattern.sub(r"[[Image:http\1jpg|No Caption]]",  wikitext)   # Above imagepattern replacements take care for external  # links, as I used them in the past to upload images to WordPress  # instead to the MediaWiki, when I intended to use them in  # articles.  #  # The new scenario is the upload to the MediaWiki and to create  # image entries with Capture Text and probaly Size Information.  # For those the image file name needs to be expanded with the  # URL to retrieve the images from the Wiki.  #  # The exported image information looks as follows:  # [[Image:Imagename.png|NNxNNpx|Capture Text]]  # Or:  # [[Image:Imagename.png|Capture Text]]  #  # The text "Imgagename.png" needs to be expanded into:  # $host/$script-path/index.php?title=File:Imagename.png  #  # Instead of Imgage the returned WikiText may contain Bild or File  # or Datei as Keyword.   # This became a NOOP, because changing the image path here  # to help pandoc to download them does not make any sense.  # Probably an image export function would make sense here.   # mstr1 = r"\[\[(Image|Bild|Datei|File):"  # mstr2 = r"([^h][^t][^t][^p].*p[n|j]g)|.*\]\]"  # mstr = mstr1 + mstr2  # imagepattern = re.compile(mstr, flags=re.MULTILINE)  # replstr = r"" + site + r"index.php?title=" + r"\2"  # wikitext = imagepattern.sub(replstr, wikitext)   # Write the result to disk.  # TODO: Enable also enviroment variables to determine HOME.  # Using ~ is nice, but quite OS dependent  if exportdir[0] == '~':  wikifile = Path.home() / exportdir.strip('~').strip('/') \  / "{0}.{1}".format(pagename, "mediawiki")  else:  wikifile = Path(exportdir) \  / "{0}.{1}".format(pagename, "mediawiki")  wikifile = wikifile.resolve()  with open(wikifile, 'w') as outfile:  print(wikitext, file=outfile)  outfile.flush()  outfile.close()  print('\nThe mediawiki file was exported to:\n' + outfile.name)   prompt = '\nDo you want to review the file? yes/no (y/n):'  pressed = askforkeypress(prompt=prompt,  keylist=['y', 'Y', 'n', 'N'],  onerror='n')  if pressed in ['y', 'Y']:  subprocess.run(["vim", wikifile])   # TODO: Consider to execute the commit into the wiki git  # htmltext = subprocess.run(["pandoc", "-f", "mediawiki", \  # "-t", "html"], input=bytearray(wikitext.encode()), \  # capture_output=True)  # print(htmltext.stdout.decode("utf-8"))  break  return   if __name__ == "__main__":  # Check command line arguments, provide help and call the functions   CREATEFLAG = False   try:  opts, args = getopt.getopt(sys.argv[1:], "hw:", ["help", "wiki"])   except getopt.GetoptError:  print(HELPTEXT)  sys.exit(2)   # Defaults  CFGSECTION = 'Default'   for opt, arg in opts:  if opt in {"-h", "--help"}:  print(HELPTEXT)  sys.exit()  if opt in {"-w", "--wiki"}:  CFGSECTION = arg   arg_names = ['pagename']  args = dict(zip(arg_names, args))   # print(args)   # Kept as inspiration for future  # ------------------------------  # Arg_list = collections.namedtuple('Arg_list', arg_names)  # args = Arg_list(*(args.get(arg, None) for arg in arg_names))   pagename = args.get('pagename')  if not pagename:  print(colored('\nERROR:', 'red'), 'Page_Name parameter is missing.')  print(HELPTEXT)  sys.exit(2)   config = configparser.ConfigParser()   configpath = Path.home() / '.config' / 'wikitools' / 'wikitools.cfg'  config.read(configpath)  sections = config.sections()  if CFGSECTION not in sections:  print(colored('\nERROR:', 'red'), 'Configuration is missing.')  print(HELPTEXT)  sys.exit(3)   export()   sys.exit(0)

~/.config/wikitools/wikitools.cfg

The code reads a configuration, which enables me to post the code without the risk of exposing my user and password for the wiki instances I use.

One default wiki can be configured and as many additional wikis as you like. With the command line paramente -w you can address the configuration section you want to use in current export.

The configuration is shared between multiple wikitools.

# # The configurations location has to be # # UserHome/.config/wikitools/ # # If the command line names no wiki section # the Default section is used. # # The command line option -w # with a parameter can be used to name a wiki, # for which a configuration section exists. # # access to a key from python # config[cfgsection]['key']  [Default] DefaultCategory = Your Category Host = Your wiki host name ScriptPath = Your wiki script path User = Your wiki user name Passwort = Your wiki password References = == Fußnoten == ExportDirectory = ~/projects/idee/author  [yourwiki] DefaultCategory = Your Category Host = Your wiki host name ScriptPath = Your wiki script path User = Your wiki user name Passwort = Your wiki password References = == Footnotes == ExportDirectory = ~/projects/idee/author

WordPress Migration

Meta data and, if I decide to use this, also the content of WordPress articles and sites, are available in the MariaDB database wp_idee.

In the context of the project it made sense to grant remote access to the MariaDB from the local area network. This is well described in the official documentation ^{( 13 )} .

root @sol:~# cd /etc/mysql/mariadb.conf.d/ root @sol:/etc/mysql/mariadb.conf.d# ls 50-client.cnf 50-mysql-clients.cnf 50-mysqld_safe.cnf 50-server.cnf root @sol:/etc/mysql/mariadb.conf.d# vim 50-server.cnf 

In file 50-server.cnf comment out the the bind-address 127.0.0.1, making it bind to the network cards addresses. In my case there is just one if these,

[...] # Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. # bind-address = 127.0.0.1 [...]

Restart of the sql server.

root @sol:/etc/mysql/mariadb.conf.d# systemctl restart mysql

Login into the sql server as root to manage users.

root @sol:/etc/mysql/mariadb.conf.d# mysql -p

MariaDB [(none)]> SELECT User, Host FROM mysql.user; +-----------+-----------+ | User | Host | +-----------+-----------+ | ninja | localhost | | root | localhost | | wiki | localhost | | wordpress | localhost | +-----------+-----------+ 4 rows in set (0.000 sec) MariaDB [(none)]> CREATE USER wpremote@'10.19.67.%' IDENTIFIED BY 'password-of-new-user'; Query OK, 0 rows affected (0.001 sec) MariaDB [(none)]> SELECT User, Host FROM mysql.user; +-----------+------------+ | User | Host | +-----------+------------+ | wpremote | 10.19.67.% | | ninja | localhost | | root | localhost | | wiki | localhost | | wordpress | localhost | +-----------+------------+ 5 rows in set (0.000 sec) MariaDB [(none)]> GRANT ALL PRIVILEGES ON wp_idee.* TO 'wpremote'@'10.19.67.%' WITH GRANT OPTION; Query OK, 0 rows affected (0.017 sec)

Remote Access with the new user.

frank @Asimov:~$ mysql -u wpremote -h sol -p wp_idee Enter password:  Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A  Welcome to the MariaDB monitor. Commands end with ; or \g. Your MariaDB connection id is 205 Server version: 10.3.31-MariaDB-0+deb10u1 Debian 10  Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.  Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.  MariaDB [wp_idee]> 

Table wp_posts

MariaDB [wp_idee]> describe wp_posts; +-----------------------+---------------------+------+-----+---------------------+ | Field | Type | Null | Key | Default |... +-----------------------+---------------------+------+-----+---------------------+ | ID | bigint(20) unsigned | NO | PRI | NULL | | post_author | bigint(20) unsigned | NO | MUL | 0 | | post_date | datetime | NO | | 0000-00-00 00:00:00 | | post_date_gmt | datetime | NO | | 0000-00-00 00:00:00 | | post_content | longtext | NO | | NULL | | post_title | text | NO | | NULL | | post_excerpt | text | NO | | NULL | | post_status | varchar(20) | NO | | publish | | comment_status | varchar(20) | NO | | open | | ping_status | varchar(20) | NO | | open | | post_password | varchar(255) | NO | | | | post_name | varchar(200) | NO | MUL | | | to_ping | text | NO | | NULL | | pinged | text | NO | | NULL | | post_modified | datetime | NO | | 0000-00-00 00:00:00 | | post_modified_gmt | datetime | NO | | 0000-00-00 00:00:00 | | post_content_filtered | longtext | NO | | NULL | | post_parent | bigint(20) unsigned | NO | MUL | 0 | | guid | varchar(255) | NO | | | | menu_order | int(11) | NO | | 0 | | post_type | varchar(20) | NO | MUL | post | | post_mime_type | varchar(100) | NO | | | | comment_count | bigint(20) | NO | | 0 | +-----------------------+---------------------+------+-----+---------------------+ 23 rows in set (0.003 sec)

Post Types

MariaDB [wp_idee]> select distinct post_type from wp_posts; +---------------+ | post_type | +---------------+ | attachment | | nav_menu_item | | page | | post | | revision | +---------------+ 5 rows in set (0.003 sec)

The post types of interest are most probably only page and post. I'm not sure about revision, but we can take a look, what's tagged as post type "revision".

The query "select post_title from wp_posts where post_type='revision';" brings repeatedly the same title, most probably the post_modified information should differ in those.

The query "select post_title from wp_posts where post_type='page';" shows only 2 pages, which are already processed into one for the new solution.

The query "select post_title from wp_posts where post_type='post';" shows every title only once, i would hope with the initial post information only. I'll not only hope, but check this before I continue.

The query "select post_name from wp_posts where post_type='post';" shows the pages unique identifiers, the last element of the pages URL. This information is very helpful to ensure that the migrated pages are named exactly the same in the new solution and are found via one redirect instruction in nginx. The redirect is important, since I will not have the date encoded in the URL, as I had it in wordpress.

Post Mime Types

Post Mime types help to select PDF, Audio and zip and Spreadsheets, which had been embedded into the posts in WordPress. The HTML posts themselves got the mime type "" in this DB, which saves space, but is irritating.

Missing Information

One Information is missing. The language or locale of the posted content. In the new solution I'll use de-DE and en-US as locales and as language information. I did not really create a lot of English pages, but this might change and the few ones I already have shall get presented correctly.

WordPress Meta Data Export

Create an export query for the HTML pages posted in WordPress. Include

• a site column with default value "Idee",
• a locale column with default value "de-DE",
• a author column with default value "Frank Siebert"

to be changed manually for the few English posts existing into

• site = "Concept" (Concept of new cognition elicitation personally thinking) replacing Idee (Idee der eigenen Erkenntnis). That's the best recursive acronym translation I found.
• locale = "en-US"

Get the earliest post date into one column and the latest post date into another column of the same row. Have one row per post.

The export query is created in a bash script, whose output can be piped into a local tab delimited file. The last line of this file needs to be deleted, since it contains an automatically saved draft, and .. (see next chapter).

wpmeta bash script:

#!/bin/bash sql="select \  min(post_date) over (partition by post_name) as post_date, \  max(post_modified) over (partition by post_name), \  max(comment_count) over (partition by post_name), \  'Idee' as site, \  'de-DE' as locale, \  'Frank Siebert' as author, \  post_name, \  post_title \  from wp_posts \  where (post_mime_type='' and post_type='post') \  order by post_date asc;"  mysql wp_idee -u wpremote -h sol -p'not-exposed-pwd' \ --default-character-set=utf8 -N -e "$sql" > ../config/migrationlist.csv

Using bash for the query and persisting the result directing the output into a file, the data in the file becomes tab delimited.

Special-Case for nginx redirect

Querying the database revealed: The post "Social Distancing und Lockdown" has for unknown reasons the url "/2021/01/26/255/". This will not be the case in the new solution, I'll not have a 255.html around there, but I'll either create a special case in the redirect for this article or rather ignore this case at all.

In the result of the final export query a corrected page_name needs to be maintained, to migrate this article correctly.

Option in Consideration

For all articles written in the Wiki and published afterwards, export.py from the wikitools will do perfect service, I hope.

But things change over time and quite a number of articles where written in WordPress, when I had no MediaWiki in place. Also some last minute typo correction, I know this, where maintained directly in WordPress after publishing.

And the article "Das SARI-Rätsel" contains an interactive java-script part, which required some authoring in html. This last point might not stay a single happenstance, as developer I might find more often a reason to extend a page functionally, or even to write the complete page directly in html as source format.

My concept does support such deviations from the standard scenario. But this is not the point here. The point is: I need an wp-export tool to generate MediaWiki files from the current WordPress pages for the migration. That's the only way to ensure that the migrated pages contain exactly what they are supposed to contain. With this I can also incorporate comments posted by me after publishing as Update Notes.

I need to leave a note about the last paragraph. For the reasons already mentioned above I made a highly manual migration. Overall content quality improved considerably during migration, even if I missed one or two typos.

A migration file created via sql from the wp_posts table ist stored as ./config/migration.csv (with spaces instead of colons) containing date, time and title columns.

wp-export.py

This was never implemented. I place this tool in the directory ./tools/ of the Authoring GIT and create an alias wpe for the single file export and probably I'll also have an wpbe command as WordPress batch-export.

Manage Meta Data

I have now migration data in a tab delimited csv available, and I need to manage publishing meta data to make sure the correct meta data is shown in every web-page, in the sitemaps and RSS-feed.

I started to implement this as a specialized dictionary implementation, but I falter to proceed in this direction. The python modules pandas and agate seem to offer fascinating power working with csv files, and I have to investigate both in much more detail in the future.

My choice fell on the module agate (Documentation: "agate 1.6.3" ^{( 14 )} ) for this implementation, since it is a smaller implementation and because I do not need any extensive statistical horsepower for the meta data stored. I use the module in version 1.6.1 as it is provided currently by Debian Stable.

I have to revise me decision. Agate makes the reading of the csv easy and provides a powerful table object, making the data accessible very nicely. But the documentation states that it returns always copies to the data structure, which I like very much, but the documentation fails to show options to update data in the table object, which I will need to do without getting new instances of the meta data table next to the singleton created for that purpose.

Installing python3-pandas

frank @Asimov:~$ sudo apt-get install python-pandas-doc python3-pandas [sudo] password for frank:  Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed:  libblosc1 libclang-cpp9 libffi-dev liblbfgsb0 libllvm9 libncurses-dev libpfm4   libtinfo-dev libz3-dev llvm-9 llvm-9-dev llvm-9-runtime llvm-9-tools   numba-doc python-odf-doc python-odf-tools python-tables-data   python3-bottleneck python3-et-xmlfile python3-iniconfig python3-jdcal   python3-llvmlite python3-numba python3-numexpr python3-odf python3-openpyxl   python3-pandas-lib python3-py python3-pytest python3-scipy python3-tables   python3-tables-lib python3-xlwt Suggested packages:  ncurses-doc llvm-9-doc python-bottleneck-doc llvmlite-doc nvidia-cuda-toolkit   python3-statsmodels python-scipy-doc python3-netcdf4 python-tables-doc   vitables python3-xlrd python-xlrt-doc The following NEW packages will be installed:  libblosc1 libclang-cpp9 libffi-dev liblbfgsb0 libllvm9 libncurses-dev libpfm4   libtinfo-dev libz3-dev llvm-9 llvm-9-dev llvm-9-runtime llvm-9-tools numba-doc   python-odf-doc python-odf-tools python-pandas-doc python-tables-data   python3-bottleneck python3-et-xmlfile python3-iniconfig python3-jdcal   python3-llvmlite python3-numba python3-numexpr python3-odf python3-openpyxl   python3-pandas python3-pandas-lib python3-py python3-pytest python3-scipy   python3-tables python3-tables-lib python3-xlwt 0 upgraded, 35 newly installed, 0 to remove and 1 not upgraded. Need to get 84.2 MB of archives. After this operation, 496 MB of additional disk space will be used. Do you want to continue? [Y/n] y

A lot of suggestions next to a lot of required packages. If I would not intend to go deeper into data analysis with python, I would stay with my initial dict-based implementation instead.

python3-pandas web references

• "pandas documentation" ^{( 15 )}
• "Add new rows and columns to Pandas dataframe" ^{( 16 )} With the most helpful explanation to insert via loc (or update via iloc) rows with:

df.loc[len(df.index)]=list(data[0].values())

• "Pandas Tutorial" ^{( 17 )}

Bioconductor

The search "derive from dataframe" had one result on startpage.com. This search result does not help me to find out, what to take special care for when I derive my own class from the dataframe class, but it relates strongly to a lot of articles I posted on my site.

• "Getting Started with Bioconductor 3.7" ^{( 18 )}

A quick check reveals - it is available in Debian just at my fingertips.

frank @Asimov:~$ sudo apt-cache search bioconductor bio-tradis - analyse the output from TraDIS analyses of genomic sequences libtfbs-perl - scanning DNA sequence with a position weight matrix q2-dada2 - QIIME 2 plugin to work with adapters in sequence data r-bioc-affy - BioConductor methods for Affymetrix Oligonucleotide Arrays r-bioc-affyio - BioConductor tools for parsing Affymetrix data files ... r-bioc-variantannotation - BioConductor annotation of genetic variants r-bioc-xvector - BioConductor representation and manpulation of external  sequences r-bioc-zlibbioc - (Virtual) zlibbioc Bioconductor package r-cran-ape - GNU R package for Analyses of Phylogenetics and Evolution r-cran-biocmanager - access the Bioconductor project package repository

I suppose I'll never have enough leisure time to dig into everything I'm interested in. And it is written in R, a programming language I would like to learn as well.

I have to remind me from time to time that I'm doing this to learn and not to to prove I can do this implementation in very short time. More reading, less coding, take your time and it will take less time.

MediaWiki to Plain HTML Conversion

You are right, if you make the assumption that I started much earlier writing Python code. But we are now finally at the point where first parts of the final Python implementation can be used in the final setup.

I spare every evolutionary step of the Python code development. All following code is in the most recent state.

Commit-msg Nessage Hook

''Originally I wrote an commit-msg hook directly as executable Python program. But the message hook shown next is a bash scriot.

~/projects/idee/config/hooks/commit-msg

#!/bin/bash /usr/bin/python3 generator/commitmsg.py $1

The parameter $1 is the commit message, for which the solution does use a modified template.

Commit Message Template

The commit message template provides the possibility to define some meta data to taken into the content and to steer apart different content creation options.

~/projects/idee/config/commit-message

  # Overwrite values if neccesary, based on https://ogp.me/ # pdf:draft=false # og:locale=de-DE # og:site_name=Idee # article:author=Frank Siebert #

Note: There is a significant empty line as the first line.

Probably I will eliminate either the og:site_name or the og:locale line in future. I'm not yet sure.

Main Program: commit-msg.py

You might have seen it above, this is the Python program called by the message hook bash script. The commit message is passed on as parameter.

The commit-msg.py registers workers to take care for specific the commit message entries identified by a regex match at a dispatcher class.

~/projects/idee/generator/commit-msg.py

"""Website Generator - "pandoc, fs-commit-msg-hook 1.0".  @author: Frank Siebert @license: https://creativecommons.org/publicdomain/zero/1.0/deed.en @date: 2022-03-15  https://wiki.frank-siebert.de/inst/Replacing_Wordpress https://idee.frank-siebert.de/article/replacing-wordpress.html  Website Generator uses Beautiful Soup, Pandoc and GIT to manage authoring in *.wikimedia files and to convert those into: * plain html pages, one per wikimedia file as article * PDF files, one per wikimedia file for article download * a Web site portal pages by injecting the portal into the plain html pages  Website Generator generates as additional portal assets: * sitemap.xml * feed.xml * ...  Website Generator works with Python 3 and up. It works better if lxml and/or html5lib is installed, as Beautiful Soup states it runs better then. """ # Systen Imports import sys import getopt  from termcolor import colored  from gitmsgdispatcher import GitMsgDispatcher from mwworker import MwWorker from pdfworker import PdfWorker from plainworker import PlainWorker  # Ask for a key press and return the pressed key, if it is part of the keylist. # In case of errors return the value of onerror, to enable the caller to # decide on the most convinient way to preceed.   if __name__ == "__main__":   HELPTEXT = 'Usage: commit-msg \'message-file\'\n'\  '\n'\  'message-file The commit message file with the list of files\n'\  ' to be processed.\n'\   try:  opts, args = getopt.getopt(sys.argv[1:], "h:", ["help"])   except getopt.GetoptError:  print(HELPTEXT)  sys.exit(2)   for opt, arg in opts:  if opt in {"-h", "--help"}:  print(HELPTEXT)  sys.exit()   arg_names = ['message-file']  args = dict(zip(arg_names, args))   # print(args)   # Kept as inspiration for future  # ------------------------------  # Arg_list = collections.namedtuple('Arg_list', arg_names)  # args = Arg_list(*(args.get(arg, None) for arg in arg_names))   messagefile = args.get('message-file')  if not messagefile:  print(colored('\nERROR:', 'red'), 'message-file parameter is missing.')  print(HELPTEXT)  sys.exit(2)   mwworker = MwWorker(r".*(new file|modified).*author[/].*\.mediawiki")  plainworker = PlainWorker(r".*(new file|modified).*plain[/].*\.html")  pdfworker = PdfWorker(r"" + PdfWorker.pdfworkitem)   disp = GitMsgDispatcher(messagefile, [mwworker, plainworker, pdfworker])   sys.exit(0)

Message Dispatcher: gitmsgdispatcher.py

The message dispatcher dispatches work-items from the git message to registered workers. Those workers then can place new work-items for later workers, to pick up work where they stopped working.

~/projects/idee/generator/commit-msg.py

""" GitMessageDispatcher with MsgWorker base class.  @author: Frank Siebert @license: https://creativecommons.org/publicdomain/zero/1.0/deed.en @date: 2022-03-15  Instantiate the GitMessageDispatcher with the message and with specialized workers. The workers provide a pattern matching the lines they claim for work. """ import re from pathlib import Path from pubmetadata import PubMetaData from sitemap import SiteMap from archive import Archive from rssbuilder import RSSBuilder from idxbuilder import IDXBuilder   class GitMsgDispatcher:  """  Dispatch the lines of the git message to registered workers.   Parameters  ----------  gitmessagepath : Path  Path as type str or type Path pointing to the git message.   msgworkers : List of MsgWorker  The list of message workers is used as worker queue. Workers first  in the queue get their workitems first.   Workers can return their work result to be picked up by  later workers.   The ParameterValueWorker runs allways first. Do not add  the ParamterValueWorker to the provided list of workers,  or it will run twice.   The ParameterValueWorker takes care to provide the parameter  values provided by the message in place for all workers.   When all workers finished, the sitemap is updated, the RSS  feed is updated and the index page is updated.   Returns  -------  GitMsgDispatcher.   """   def __init__(self, gitmessagepath, msgworkers):  """  Dispatch the lines of the git message to registered workers.   Parameters  ----------  gitmessagepath : Path  DESCRIPTION.  *msgworkers : List of MsgWorker  DESCRIPTION.   Returns  -------  GitMsgDispatcher.   """  self.gitmessagepath = gitmessagepath   """  Extract the relevant part of the message  """  self.worklist = []  self.parameters = ParameterValueWorker()   with open(self.gitmessagepath, 'r') as infile:  # TODO: Do Better. Latest when the git server joins the game  # Message section of most interest: "Changes to be committed"  # But we are also interested in the parameter values  # we placed ealier into the file.  # The Start helps us to find the End.  start = re.compile(r"^# Changes to be committed:")  # Next uppercase entry starts another message section  end = re.compile(r"^# [A-Z]")   started = None  for line in infile:  if self.parameters.pattern.match(line):  self.worklist.append(line)  if not started and start.match(line):  started = True  elif started and end.match(line):  started = False  if started is True:  self.worklist.append(line)  if started is False:  break  infile.close()   self.msgworkers = msgworkers   self.msgworkers.insert(0, self.parameters)  self.dispatch()   # Save changed publishing meta data, if any.  if PubMetaData.instance:  PubMetaData.instance.save()  # Generate Sitemaps (bilingual)  SiteMap().update()  # Generate Archive  Archive().update()  # Generate RSS feed (bilingual)  RSSBuilder().update()  # Generate Index Pages (English and German Version)  IDXBuilder().update()   def dispatch(self):  """  Dispatch the git message lines to registered message workers.   Returns  -------  None.   """  for worker in self.msgworkers:  print("Dispatching work to: {}".format(type(worker)))  for item in self.worklist:  if type(item) == str:  if worker.pattern.match(item):  worker.work(self, item)  if type(item) == dict:  worker_match = item[MsgWorker.task_worker_match]  if worker.pattern.match(worker_match):  worker.work(self, item)   class MsgWorker:  """  Create a worker for lines matching the pattern.   Parameters  ----------  pattern : Pattern  A regex pattern matching the lines the worker takes care for.   Returns  -------  MsgWorker.   """   # Cases to work against  CREATEPAT = re.compile(r"^#.*(modified:|new file:)")  RENAMEPAT = re.compile(r"^#.*renamed:")  DELETEPAT = re.compile(r"^#.*deleted:")   # Task types  task_worker_match = "workermatch"  task_type = "tasktype"  task_create = "create"  task_rename = "rename"  task_delete = "delete"   def __init__(self, pattern):  if isinstance(pattern, str):  self.pattern = re.compile(pattern)  elif isinstance(pattern, re.Pattern):  self.pattern = pattern  self.dispatcher = None # initialized in work method  self.item = None # initialized in work method  self.inpath = None # initialized in work method, if any  self.delpath = None # initialized in work methid, if any  self.outpath = None # initialized in work method, if any   def get_pattern(self):  """  Get the pattern for matching list items.   Returns  -------  Pattern  A regex pattern matching the lines the worker takes care for.   """  return self.pattern   def work(self, dispatcher, item):  """  Overwrite this method to implement if required.   Call the super().work() method in your new method, to get  - self.dispatcher initialized  - the inpath initialized (if any), to the file to be processed  - the delpath initialized (if any)  - the delete() or the process() method or both called, whatever applies   Parameters  ----------  dispatcher : GitMsgDispatcher  The dispatcher, which assigned the work item   item : str or dict  One matching line from the git message.  Or complex workitem added by ealier workers.   Returns  -------  None.   """  self.dispatcher = dispatcher  self.item = item   # For some workers a dictionary is passed as item  if isinstance(item, str):  filename = item[14:].strip()  self.inpath = Path(filename)   if self.RENAMEPAT.match(item):  # part filename in new and old  # filename = line[14:].strip()  # self.inpath = Path(filename)  self.delpath = None # Needs to be assigned now  self.rename()   if self.CREATEPAT.match(item):  self.process()   if self.DELETEPAT.match(item):  self.delpath = self.inpath # clear deletion request  self.delete()   if isinstance(item, dict):  self.inpath = None  self.delpath = None   if item[self.task_type] == self.task_rename:  self.rename()  if item[self.task_type] == self.task_create:  self.process()  if item[self.task_type] == self.task_delete:  self.delete()   def delete(self):  """  Overwrite this method to implement the actual work to be done.   The method is called by super.work(), if the message line is  about a rename or a deletion.   Since renames might go along with additional content change, deletion  and re-processing take place both in that case.   The path to the resource named in the message is available via  self.delpath   Depending on the type of content, more than just deleting the  file might be required.   Parameters  ----------  None   Returns  -------  None.   """   def process(self):  """  Overwrite this method to implement the actual work to be done.   The method is called by super.work(), if the message line is  about a rename or a new file.   Since renames might go along with additional content change, deletion  and re-processing take place both in that case.   The path to the resource named in the message is available via  self.delpath   Parameters  ----------  None   Returns  -------  None.   """   def rename(self):  """  Rename by delete and process.   Since we do not know, whether next to the rename additional  changes were applied, deletion and recreation is savest.  """  self.delete()  self.process()   class ParameterValueWorker(MsgWorker):  """  The ParameterValueWorker reads parameter value pairs.   Example of a line with parameter value pair:  # article:author=Firstname Lastname   These parameters in the git message allow the injection  of values for metadata, which would be otherwise not available.   Other workers can access the values dictionary via:  dispatcher.parameters.values   Parameters  ----------  super: MsgWorker  The ParameterValueWorker is derived from the MsgWorker.   Returns  -------  ParameterValueWorker.   """   def __init__(self, pattern=r"^#.*="):  super().__init__(pattern)  self.values = {}   def work(self, dispatcher, item):  """."""  super().work(dispatcher, item)   if item.count('=') == 1:  lineparts = item.rpartition('=')  self.values.update(  {lineparts[0].strip('#').strip():  lineparts[2].strip()}  )   if __name__ == "__main__":  pass