MP-WP Automated Testing Proposal and VPatch

The following is presented for the consideration of the lords and ladies of tmsr.

Automated End-To-End Testing

Since mp-wp is a web application consisting of a frontend and a backend, my proposal is to handle testing via end-to-end tests run in the browser itself. This will unfortunately require the operator to run the tests on a toilet box as the browser automation tool, Selenium WebDriver runs on Java and will not work with earlier vintages of Firefox (I initially tested with version 54 of Firefox and found myself having to install version 60 to get anything working). However, I believe this to be an acceptable temporary trade-off for the following reasons:

  • MP-WP is a gnarly mix of PHP, JavaScript, HTML, and CSS code. Unit tests can assert correctness of individual PHP and Javascript functions but one careless snip in a CSS file could result in a broken UI that no unit test would catch. End-to-end tests will not only insure against breakage from HTML/CSS changes, but will also allow for test coverage of critical code paths without first fully unwinding the codebase and fitting it into one's head.
  • These e2e tests are intended to be temporary, and only remain while mp-wp sweats away the pounds. They can be abandoned completely once the codebase is in its desired minimal form.
  • Only the tests need to be run on a polluted box, the requirements for running mp-wp are not affected.

I've put together a genesis patch that contains a small POC spec for the basic posting functionality in mp-wp. The spec is written as:


const { LoginPage, DashboardPage, PostsPage, EditPostPage } = require('../pages');

describe('Posts', () => {
  beforeAll(() => {
    const loginPage = new LoginPage();
    loginPage.navigate();
    loginPage.logIn();
  });

  afterAll(() => {
    const dashboardPage = new DashboardPage();
    dashboardPage.navigate();
    dashboardPage.logOut();
  });

  it('Can navigate to and display the Posts page', () => {
    const dashboardPage = new DashboardPage();
    dashboardPage.followNavLinkTo('Posts', 'Edit Posts');

    const postsPage = new PostsPage();

    expect(postsPage.pageTitle.isExisting()).toBe(true);
    expect(postsPage.postsTable.isExisting()).toBe(true);
  });

  it('Can create and publish a post', () => {
    const dashboardPage = new DashboardPage();
    dashboardPage.followNavSubMenuLinkTo('Posts', 'Add New', 'Add New Post');

    const postTitle = 'Bitcoin Declaration Of Sovereignty';
    const postBody = '<p>When in the course of human events, it becomes necessary for one person to dissolve the political bands which have connected them with the human herd, and to assume among the powers of the world the separate and equal station to which the laws of nature entitle them, a modicum of respect to their own intelligence requires that they should declare the causes which impel them to the separation.</p>';

    const editPostPage = new EditPostPage();
    editPostPage.editAndPublishPost({ postTitle, postBody });

    expect(editPostPage.page.$('#message*=Post published').isExisting()).toBe(true);

    dashboardPage.navigateToBlog();

    expect($(`a=${postTitle}`).isExisting()).toBe(true);
    expect($('.entry').getHTML(false).trim()).toEqual(postBody);
  });
});

Run the tests from the root of the project with:


./node_modules/\@wdio/cli/bin/wdio.js

This will execute the runner which will look for test files in tests/specs/ and run any it finds. This will look something like this, along with output in your console on the status of the test.

After the initial setup, writing the test didn't take very long. If this seems like a worthwhile enough approach I'll follow up with another post outlining the full set of tests to be written. After receiving input/feedback on the list, I'll proceed to write and publish the remaining tests as additional patches on this tree.

Patch and Signature

mp-wp-tests_genesis.vpatch
mp-wp-tests_genesis.vpatch.billymg.sig

Note: After pressing you will need to run npm install from the project directory in order to pull down just under 100mb of depshits in order for this to work. Comments welcome here or in #trilema.

———

5 Responses to “MP-WP Automated Testing Proposal and VPatch”

  1. Pretty clever use for the animation gif, huh!

    I don't know that/if anyone uses the selenium thing (or java, for that matter) -- I certainly never heard of it before. Can you explain to me how to specify tests in a portable format, so I can help round out your tester without having to load all that dependency tree ?

  2. billymg says:

    @Mircea Popescu

    > Can you explain to me how to specify tests in a portable format, so I can help round out your tester without having to load all that dependency tree ?

    If you are ok with me using this heathen tester concoction strictly as demolition scaffolding while I reduce mp-wp to its essentials then a spec written as Describe / It blocks would be really helpful. For example:

    Describe: "Media Library"
    It: "Can display a list of uploaded media in a table"
    It: "Can upload a new item to the media library"
    It: "Can view an item from the media library"
    It: "Can edit an item from the media library"

    etc...

    If I had a list like that which covered all the parts of mp-wp you would like to keep around I could immediately get started on the grunt work of writing those tests. Once full coverage is achieved, I could then get to the more fun part of razing large sections of the codebase, confident that so long as the tests continue to pass we haven't accidentally blown up something useful.

    After mp-wp is in a desirable state we simply abandon these tests all together, and think of a testing solution that doesn't involve Selenium, Java, and mess of assorted JS libs. I feel this is the fastest way to the desired outcome, and in this sense exploits the heathen tools without relying on them.

  3. Interestingly -- I actually used "selenium" at one point, decade ago; even wrote a CL frontend for it. It's a tool for "headless" operation of graphical WWW browsers, pretty handy for auto-"cheating" various webisms, grabbing text, breaking "captcha"isms, etc.

  4. billymg says:

    @Stanislav: Neato, hadn't thought to use for this purpose but that is good to know.

    While researching alternatives for this project I learned about Marionette. It looks as though that would be the raw/low-level way to interact with Firefox, but requires writing own frontend. It's something I want to come back to later when I have more time or more hands.

  5. [...] billymg Do you read the logs? You should probably read the logs. « MP-WP Automated Testing Proposal and VPatch [...]

Leave a Reply