featkpr

Who gets which answer, and the tests that prove it.

For every route, featkpr lists who must get through and who must be refused, writes a test for each, runs it, then sends it again as the wrong user. Four screens show it, in the order an engineer reads them.

Screens of featkpr's app run on sample data, so the numbers inside them are examples, not BookStack's. A screen marked "recorded from BookStack" replays featkpr's own records of BookStack's run of 26 Sep.

Every scenario of a route, as a row of pins #

A route does several things, and each needs its own permission. featkpr reads the checks from the code and lists every combination of permissions with the answer BookStack must give. Each combination is a scenario: a pin.

Home/homesample data

  1. The route, as the code names it.
  2. A row per thing the route does. Restoring also needs page-update.
  3. A pin per scenario. Filled: a test covers it. Hollow: no test yet.
  4. Covered, of all the scenarios in the row.
  5. Amber: two scenarios a person must decide before any test is written. They open as questions.

The whole tree of a feature, with the real screen of each row, is the Tree screen; a pin opens its Scenario.

The test file featkpr writes for one pin #

Each scenario becomes a pytest file from a template, never by hand. The file says who the test's user is, who the wrong user is, and which user is the control. Read the numbered lines: they are the whole idea of the check.

POST /books/{bookSlug}/convert-to-shelf“Refused with book-update, without the other 4” · 170 lines · the lines that matter
31Scenario 003552589969805e9a570822e4874eb278f210a8e5ea0adee02e6bc47f8320bb, level 1 (rendered from its steps):4    Given permission(book-update) and not permission(book-create-all) and not permission(book-delete) and not permission(book-view) and not permission(bookshelf-create-all).5    1. L1: send POST /books/{bookSlug}/convert-to-shelf; expect refused6    Then refused.9 lines not shown16SCENARIO_ID = "003552589969805e9a570822e4874eb278f210a8e5ea0adee02e6bc47f8320bb"172ACTOR = None  # no seeded role satisfies the predicate: the test makes a role holding exactly ACTOR_PERMISSIONS, and a user in it18ACTOR_PERMISSIONS = ['book-update-all']193FLIP_ACTOR = "admin"  # role: Admin — the test must fail as this user204CONTROL = "admin"  # role: Admin — sends the same request and must get through3 lines not shown24EXPECT = 'refused'1 line not shown26REACHED = (302, '{base}/shelves/[a-z0-9-]+')  # BookStack's success answer: status, Location of a redirect8 lines not shown35HIDDEN = 'Book not found'  # errors.book_not_found: ACTOR cannot view what the route names; BookStack hides it103 lines not shown139def test_003552589969805e(actor_session, fresh_page, env, browser, flip):1405    if flip and FLIP_ACTOR is None:141        pytest.skip("every seeded user satisfies the predicate: nobody to flip to")16 lines not shown158        if flip:1596            page = actor_session(FLIP_ACTOR, FLIP_ACTOR)160        else:161            page = actor_ctx = _sign_in(browser, base, marker + "-actor@m6.test", FIXTURE_PASSWORD)1627        r = _send(page, base, url, form, files)163        _expect_refused(page, base, r)164        control = actor_session(CONTROL, CONTROL)  # the same request gets through: the refusal was the permission's165        _expect_reached(_send(control, base, url, form, files), base, v)5 lines not shown
  1. The scenario in words: one permission granted, four withheld, and the answer BookStack must give. Here: refused.
  2. The test's own user. No user our seed made holds exactly this permission, so the test makes a role and a user for itself.
  3. The wrong user. This test expects a refusal, so its wrong user is the Admin, who must get through.
  4. The control: the Admin sends the same request and gets through, so a refusal is the permission's and not a broken page.
  5. The wrong-user run signs in only as users our seed made. When none of them can play the wrong user, the test is skipped: that is where the unchecked tests come from.
  6. In the wrong-user run, the same request goes out as the wrong user…
  7. …and the answer must still be a refusal (the next line checks it). As the Admin it is not, so the test fails, as it must.
Read the whole file, all 170 lines
1"""Emitted by ae tests emit, template 'bookstack/access'. The cache block is the tool's; a regeneration keeps hand edits elsewhere.2 3Scenario 003552589969805e9a570822e4874eb278f210a8e5ea0adee02e6bc47f8320bb, level 1 (rendered from its steps):4    Given permission(book-update) and not permission(book-create-all) and not permission(book-delete) and not permission(book-view) and not permission(bookshelf-create-all).5    1. L1: send POST /books/{bookSlug}/convert-to-shelf; expect refused6    Then refused.7"""8import base649import re10import urllib.parse11import uuid12 13import pytest14 15# --- cache block: the only part the tool rewrites ---16SCENARIO_ID = "003552589969805e9a570822e4874eb278f210a8e5ea0adee02e6bc47f8320bb"17ACTOR = None  # no seeded role satisfies the predicate: the test makes a role holding exactly ACTOR_PERMISSIONS, and a user in it18ACTOR_PERMISSIONS = ['book-update-all']19FLIP_ACTOR = "admin"  # role: Admin — the test must fail as this user20CONTROL = "admin"  # role: Admin — sends the same request and must get through21PREDICATE = '(permission(book-update) and not permission(book-create-all) and not permission(book-delete) and not permission(book-view) and not permission(bookshelf-create-all))'22METHOD = 'POST'23ROUTE = '/books/{bookSlug}/convert-to-shelf'24EXPECT = 'refused'25API = False26REACHED = (302, '{base}/shelves/[a-z0-9-]+')  # BookStack's success answer: status, Location of a redirect27FORM = {}28FILES = {}29TIMEOUT_MS = 1000030FIXTURE_PASSWORD = 'm6-test-password'31T_PERMISSION_NOTICE = 'You do not have permission to access the requested page.'  # errors.permission32T_PERMISSION_JSON = 'You do not have permission to perform the requested action.'  # errors.permissionJson33T_API_ACCESS = 'The owner of the used API token does not have permission to make API calls'  # errors.api_user_no_api_permission (ApiAuthenticate.php:35-39)34REFUSED_BY = 'hidden'  # ACTOR cannot view what the route names: BookStack hides it35HIDDEN = 'Book not found'  # errors.book_not_found: ACTOR cannot view what the route names; BookStack hides it36# --- end cache block ---37 38NOTICE_RE = r'class="notification neg"[^>]*role="alert">.*?<span>{}</span>'39FIXTURE_TIMEOUT_MS = 30000  # the fixtures and the clean-up: never the step's own bound, so a step's timeout is its own40 41 42def _api(admin, method, path, body=None):43    r = admin.fetch("/api/" + path, method=method, data=body, timeout=FIXTURE_TIMEOUT_MS)44    assert r.status in (200, 201, 204), f"fixture: {method} /api/{path} answered {r.status}: {r.text()[:200]}"45    return r.json() if r.status != 204 and r.body() else {}46 47 48def _send(page, base, url, form, files):49    """The request, as the signed-in user: nothing followed, so BookStack's first answer is judged.50 51    Every web request, a GET too, is preceded by a load of the home page: an actor's session is shared by52    every test of the run, so a notice an earlier request left pending (a refusal nobody followed) would53    otherwise show on the page after this one, and a 302 to the home page that BookStack's session never54    saw (a proxy, a gateway) would pass as the refusal. The load takes that notice up first."""55    if API:56        return page.request.fetch(url, method=METHOD, headers={"Accept": "application/json"}, max_redirects=0,57                                  timeout=TIMEOUT_MS)58    home = page.request.get(base + "/", max_redirects=0, timeout=TIMEOUT_MS)  # takes up any notice still pending59    if METHOD == "GET":60        return page.request.fetch(url, method=METHOD, max_redirects=0, timeout=TIMEOUT_MS)61    m = re.search(r'<meta name="token" content="([^"]+)"', home.text())62    assert m, f"no CSRF token on the home page (it answered {home.status})"63    fields = {"_token": m.group(1), **({} if METHOD == "POST" else {"_method": METHOD}), **form}64    if files:65        return page.request.post(url, multipart={**fields, **files}, max_redirects=0, timeout=TIMEOUT_MS)66    return page.request.post(url, form=fields, max_redirects=0, timeout=TIMEOUT_MS)67 68 69def _expect_refused(page, base, r):70    """BookStack's refusal, measured: 302 to the home page, which shows the permission notice; on the API,71    403 with the permission message — or, where REFUSED_BY says the API gate refuses first, its own message.72    Anything else (a 500, a 404, a 419, a timeout, another 403) fails."""73    if HIDDEN is not None:  # an entity the user cannot view is hidden: 404 with its own not-found answer74        assert r.status == 404, f"REFUSED (hidden) is BookStack's 404; this answered {r.status}"75        if API:76            err = r.json().get("error")77            msg = err.get("message") if isinstance(err, dict) else err78            assert msg == HIDDEN, f"a 404 that is not BookStack hiding the entity ({HIDDEN!r}): {msg!r}"79            return80        assert re.search(r"<h1[^>]*>\s*" + re.escape(HIDDEN) + r"\s*</h1>", r.text()), \81            f"a 404 that is not BookStack hiding the entity ({HIDDEN!r})"82        return83    if API:84        assert r.status == 403, f"REFUSED on the API is BookStack's 403; this answered {r.status}"85        err = r.json().get("error")86        msg = err.get("message") if isinstance(err, dict) else err87        want = T_API_ACCESS if REFUSED_BY == "api-access" else T_PERMISSION_JSON88        assert msg == want, f"a 403 that is not the {REFUSED_BY} refusal: {msg!r}"89        return90    assert r.status == 302, f"REFUSED is BookStack's 302 to its home page; this answered {r.status}"91    where = r.headers.get("location")92    assert where == base, f"REFUSED redirects to {base}; this went to {where!r}"93    home = page.request.get(where, max_redirects=0, timeout=TIMEOUT_MS)94    assert home.status == 200, f"the home page after the refusal answered {home.status}"95    assert re.search(NOTICE_RE.format(re.escape(T_PERMISSION_NOTICE)), home.text(), re.S), \96        "the home page after the redirect shows no permission notice"97 98 99def _expect_reached(r, base, v):100    """BookStack's success answer for this route: the status, and where a success redirects to."""101    status, where = REACHED102    assert r.status == status, f"REACHED is {status} here; this answered {r.status}"103    if status == 200:  # an empty 200 is a broken page, never BookStack letting the user through104        assert r.body().strip(), "REACHED is BookStack's page; this answered 200 with an empty body"105    if where is not None:106        loc = r.headers.get("location") or ""107        want = where.format(base=re.escape(base), **{k: re.escape(str(x)) for k, x in v.items()})108        assert re.fullmatch(want, loc), f"REACHED redirects to {want}; this went to {loc!r}"109 110 111def _sign_in(browser, base, email, password):112    """A user the test made, signed in through BookStack's login form. The context stands for the page:113    its `request` carries the session cookie, and that is all `_send` uses."""114    ctx = browser.new_context(base_url=base)115    login = ctx.request.get(base + "/login", timeout=FIXTURE_TIMEOUT_MS)116    m = re.search(r'name="_token" value="([^"]+)"|<meta name="token" content="([^"]+)"', login.text())117    assert m, f"fixture: no CSRF token on the login page (it answered {login.status})"118    r = ctx.request.post(base + "/login", form={"_token": m.group(1) or m.group(2), "email": email,119                                                "password": password}, max_redirects=0, timeout=FIXTURE_TIMEOUT_MS)120    back = (r.headers.get("location") or "").endswith("/login")  # a refused login goes back to the form121    home = ctx.request.get(base + "/", max_redirects=0, timeout=FIXTURE_TIMEOUT_MS)122    assert r.status == 302 and not back and re.search(r'<form[^>]*action="[^"]*/logout"', home.text()), \123        f"fixture: {email} did not sign in (the login answered {r.status} to {r.headers.get('location')!r})"124    return ctx125 126 127def _cleanup(admin, made, marker):128    """Delete what this test made: its page, its fixtures, and every book, shelf, user and role named129    with its marker."""130    for path in reversed(made):131        admin.delete("/api/" + path, timeout=FIXTURE_TIMEOUT_MS)132    for kind, name in (("books", "name"), ("shelves", "name"), ("users", "name"), ("roles", "display_name")):133        r = admin.get(f"/api/{kind}?count=100&filter[{name}:like]=%25{marker}%25", timeout=FIXTURE_TIMEOUT_MS)134        for item in (r.json().get("data", []) if r.ok else []):135            admin.delete(f"/api/{kind}/{item['id']}", timeout=FIXTURE_TIMEOUT_MS)136 137 138@pytest.mark.scenario(SCENARIO_ID)139def test_003552589969805e(actor_session, fresh_page, env, browser, flip):140    if flip and FLIP_ACTOR is None:141        pytest.skip("every seeded user satisfies the predicate: nobody to flip to")142    base = env["base_url"]143    marker = "ae-" + uuid.uuid4().hex[:12]144    ctx = browser.new_context(base_url=base, extra_http_headers={"Authorization": "Token " + env["api_token"], "Accept": "application/json"})145    admin, made, v = ctx.request, [], {}146    actor_ctx = None147    try:148        actor_role = _api(admin, "POST", "roles", {"display_name": "ae " + marker + " actor", "permissions": ACTOR_PERMISSIONS})149        made.append("roles/" + str(actor_role["id"]))150        actor_user = _api(admin, "POST", "users", {"name": "ae " + marker + " actor", "email": marker + "-actor@m6.test", "password": FIXTURE_PASSWORD, "roles": [actor_role["id"]]})151        made.append("users/" + str(actor_user["id"]))152        book = _api(admin, "POST", "books", {"name": "ae " + marker})153        made.append("books/" + str(book["id"]))154        v["bookSlug"] = book["slug"]155        url = base + ROUTE.format(**v)156        form = {k: s.format(marker=marker, **v) for k, s in FORM.items()}157        files = {k: {"name": n, "mimeType": t, "buffer": base64.b64decode(b)} for k, (n, t, b) in FILES.items()}158        if flip:159            page = actor_session(FLIP_ACTOR, FLIP_ACTOR)160        else:161            page = actor_ctx = _sign_in(browser, base, marker + "-actor@m6.test", FIXTURE_PASSWORD)162        r = _send(page, base, url, form, files)163        _expect_refused(page, base, r)164        control = actor_session(CONTROL, CONTROL)  # the same request gets through: the refusal was the permission's165        _expect_reached(_send(control, base, url, form, files), base, v)166    finally:167        if actor_ctx is not None:168            actor_ctx.close()169        _cleanup(admin, made, marker)170        ctx.close()
A test featkpr wrote for the sample run, quoted as stored. It calls BookStack directly through Playwright's request API; tests do not drive a browser yet.

497 tests written so far.

What ran, test by test #

The Runs screen opens on the latest run's verdict. Open any test and it shows what was sent, as whom, and what BookStack answered: as the test's own user, as the control, and in the wrong-user run.

Runs/runsrecorded from BookStack, 26 Sep

  1. Every test of that run passed.
  2. In the app's own words: how many failed as the wrong user, as they must, how many are unchecked, and that none failed.
  3. The runs, newest first; the wrong-user run sits on top of the run it checks.
The Runs screen shows the run of 26 Sep, 11:17 UTC: an earlier run of that day, on the same commit. The counts below are the later run's, 17:42 UTC.

Two tests, opened #

A test the wrong-user run proves

Runs, one test openedsample data

  1. As the test's own user, an Editor: BookStack answers 302 to the new shelf, as expected.
  2. The control, the Admin, sends the same request and gets through.
  3. The wrong-user run, as a Content-only reader: BookStack sends them to its home page instead (localhost:8099 is the test copy's own address). The test fails, as it must.

A test that stays unchecked

Runs, one test openedrecorded from BookStack, 26 Sep

  1. A test that expects access: the Admin exports a book through the API and gets through.
  2. Its wrong-user run: no user our seed made lacks this permission, so there is nobody to send it as. It is skipped.

276 of 311 tests failed as the wrong user, as they must: 49 refused to a user without the permission, 216 let through for the Admin, 7 that end on a mail BookStack sends (a password reset, an invite, a comment or page notification), and 4 on another kind of answer. 35 are unchecked: they all expect access, and the wrong-user run signs in only as users our seed made; none of them lacks the permission. None got through as the wrong user when it should not have.

When a test breaks, featkpr says why #

A scenario keeps every run of its tests, step by step. When a step fails, featkpr sorts the failure before anyone reruns it: a route that moved or a label that changed is rewritten into the test and reported. A likely bug is never rewritten away; it waits for a person.

Scenario/s/…sample data

  1. The steps, each with the screen it opens. This sample has none captured, and the frames say so.
  2. Every run, a row: pass or fail on each step.
  3. What changed: the route moved, a label changed. The test is rewritten and the change reported.
  4. A likely bug. Nothing is rewritten; it waits for a person.

What a test changed in the app (rows written, mail sent) is not in a BookStack run yet. Effects and OpenTelemetry on BookStack's runs, on the plan for this week (effects recorded on BookStack's run of 28 Sep at its development tip, not shown here yet; OpenTelemetry measured on one capture run, not on every run yet).

See the wrong-user check run on BookStack #

A live walkthrough of these screens on BookStack's real run, and what the check would need for your roles.

Ask for a private demo

Open original

↑ ↓ to move, Enter to open, Esc to close