33 releases 33 tags

RSS feed

• v0.13.0 fa479de12a

  Compare

  v0.13.0 Stable

  dawid released this 2026-11-03 19:59:06 +00:00 | 130 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Cookie banners dismissed on their own: a website job's Dismiss cookie banners (on for new
    jobs, off for those saved before) has its browser answer OneTrust, Cookiebot, Usercentrics,
    Didomi, Quantcast (TCF), TrustArc, Borlabs and Klaro with "Reject all", else "Only necessary",
    and a button of exactly such a text (German, English, Polish) in a dialog or a floating banner
    that speaks of cookies; in the steps and the walk, in frames and shadow roots, and in the
    recorder's browser, which records no step of it. Nothing else is clicked; a recorded click on a
    banner is optional then.
  • Approvals in an app: the step {"action": "approve", "until": {"url", "selector"}, "label", "wait"} shows Needs you with its label and notifies the job's person, then waits until the
    page shows until (an address's pattern, an element, or both) that the approval went through,
    or fails after its wait; Stop ends it, and a page that shows until already asks nothing.
  • Handing over to the person: the step {"action": "handover", "label"} stops the run as a page
    asking for a person does, keeping the browser's session as it was there: Continue in the
    browser opens the page, the person does what only they can (a passkey), and Done runs the
    job, whose kept session skips the sign-in. Its optional until skips it when the page shows
    it already; a recipe with a handover needs its start. Approvals and handovers end the sign-in
    a kept session skips.
  • Codes from the phone: the step {"action": "ask-code", "selector", "label", "wait"} pauses
    the run at a code Harvester cannot read itself (an SMS, a call, an authenticator it has no
    secret of): the job shows Needs you with a field Enter the code from … on the jobs page
    and notifies its person, types the code sent from there into the field and goes on; after the
    wait (10 minutes by default, 30 s to 30 minutes) the run fails cleanly, and Stop ends the wait.
    The code goes only to the job's own run while it asks (POST /api/jobs/<id>/code) and is never
    logged or kept. The recorder's Steps list offers code from the phone? on a typed code, and
    Test the recipe asks for the code below the steps.
  • Exports a site prepares first: a level's or the file links' "wait_for": {"selector", "timeout", "reload"} looks for the element of the ready file on each page (reloading it every
    reload seconds if asked) for up to 30 minutes, then reads and downloads its links as before,
    clicked ones too.
  • Choices: the step each goes through every option of a <select> or a custom dropdown, or a
    fixed list of values (every account, year, child), doing the steps after it and the walk for
    each, the choice's text {choice} in the file names, tags, key and addresses. The recorder's
    Steps list offers Repeat for each option on a recorded choice, and Test the recipe lists
    the choices it goes through.
  • JSON listings asked for with POST: a form's or a JSON body (GraphQL with its variables), whose
    pages are counted in the body or chained by a cursor the answer gives (endCursor with
    hasNextPage, nextCursor, an answer's next page's address); the headers the page sent that
    matter, a token that changes read when the job runs from a cookie, a <meta> or the page's
    storage, never written into the recipe. The Network panel lists POST answers too, with a
    GraphQL operation's name, and proposes all of it from one.
  • The recorder's Network panel shows an answer's content: Preview beside each one, its JSON
    indented (the first 20 000 characters).
  • File links that are clicked ("click": true) give the file also when the click opens a tab
    or popup that shows it (a PDF in the viewer, a picture) or starts its download, or takes its
    own tab to it; checked as any download, never a web page, and the tab goes back to the listing.
  • Levels to follow that are clicked ("click": true): rows, cards or buttons without an address
    that open an entry. A run clicks each, reads the next level where it lands (another address, a
    new tab or popup, a single-page app's route, a dialog) and returns to the listing (Back,
    Escape, else the listing opened again with its loading of more); a click that fails is a gap.
  • The recorder picks elements without an address (buttons, links to javascript:, rows and cards
    a script opens) as file links or links to follow that are clicked: their pattern proposed on
    their texts, the panel saying the job clicks them, Click in the browser to switch it; Test
    the recipe walks such levels by clicking and lists clicked file links by their text.
  • Pages saved as files: the file links' "save": "pdf", "html" or "markdown" opens each page
    they lead to in the job's browser (also what an element clicked shows), settles and scrolls it,
    and saves it: Chromium's print (A4), one HTML file with its stylesheets, pictures and frames
    inside (Chromium's MHTML snapshot, its resources as data: addresses), or its main content as
    Markdown (Readability's way, on BeautifulSoup). Named by {title}, with the extension; checked
    as a download (the job's file types must take it, markdown a new one). A recipe's own
    "save" saves its start page; the template Save a page does it for the pages sent to an
    on-demand job (Fetch a URL…, Send to Harvester). The recorder's file links pick has Save
    the pages as, and Test the recipe lists the pages it would save.
  • Files a page only shows: the file links' "capture": ["application/pdf", "image/*"] keeps the
    files of those types each page loads (a viewer's PDF, pictures), embeds (<embed>, <object>,
    frames: fetched with the browser's cookies) or holds as blob: addresses (read in the page); at
    most 20 a page and 64 MB each, checked as downloads; a page whose files were all placed is not
    opened again.
  • Pictures picked as file links (img, picture) are taken at their full size: the largest
    candidate of srcset and <source>, an attribute naming the full size, the lazy-loading ones
    (data-src, data-original, …) over a placeholder src; in the browser, over HTTP and in the
    recorder.

  Changed

  • A level's pick tells a link's address apart from # and javascript: (no address).
  • Releases through the shared action nextcloud-app-release of dawid/ci-actions: the release's
    text is the version's section of this changelog, and a tag without one fails.

  Downloads

  • harvester-0.13.0.tar.gz
    0 downloads · 2026-11-03 20:11:59 +00:00 · 1 MiB
  • signature.txt
    0 downloads · 2026-11-03 20:11:59 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads