33 releases 33 tags

RSS feed

• v0.14.0 ff26738f73

  Compare

  v0.14.0 Stable

  dawid released this 2026-11-04 16:47:13 +00:00 | 27 commits to main since this release

  Signed by dawid

  GPG key ID: 4131A40E8F612CB0

  Added

  • Tags from lists: a job's tag that is a list value alone ({tags}) gives a tag per
    entry (cleaned, without repeats, 20 at most per file; the log says how many more were left
    out); elsewhere a list's entries are joined with a comma. gallery-dl's lists of texts are kept
    ({tags}) and its dicts of them become one each ({tags_general}, {tags_artist}), yt-dlp's
    {tags} and {categories} too; the form offers each source's under Tags.
  • Values of the pages a website job opens: {page_title}, {site_name}, {page_date}
    (with {page_year}, {page_month}, {page_day}) and {page_lang}, from <meta>, JSON-LD
    and <time>, read once per page (one script in the browser, the parsed page over HTTP) and
    passed down to the next levels and the file links. A level's page_values read more on the
    page its links open, and a value with "many": true is a list (tag chips, cleaned of # and
    counts).
  • Recorder: Pick tags beside a level's links (a chip in a row or card), Pick a value on
    its pages and Pick tags on its pages on a page a level's links open (the chip and those
    like it outlined); each level lists the pages it opened with their values, and Test the
    recipe shows each file link's values.
  • Save to in a job's form: the person's files (as before) or Harvester's private
    library, in the app's own storage per person, which Nextcloud's Files, Photos, search,
    WebDAV and sync clients never see, with the admin's limits per person (10 GB by default) and
    in all. A file deleted in the library is fetched again as one deleted in Files (Download
    deleted files again).
  • The private library is encrypted, a hidden space for any file: an X25519 key pair per
    person, its private key wrapped with the library password (scrypt) and a recovery key shown
    once at setup (typed back to confirm); each file's content in 64 KiB chunks of AES-256-GCM
    (ranges decrypt their chunks alone; a chunk changed, swapped or cut off is found), its key
    sealed to the public key, its name, folder, size, type, labels and thumbnail encrypted too.
    Jobs, uploads and moves write while it is locked; reading needs it unlocked, the private key
    in the server's memory only, dropped at lock, after the person's minutes and at a restart.
    Library on the page: setup, unlock, recovery, folders, thumbnails (Pillow), a viewer,
    search by name and label, uploads in pieces (drag and drop), Rename, Move, Delete, and a
    cookie (HttpOnly, SameSite=Strict) or per-file tickets instead of a token in addresses.
  • Move to the private library in Files' actions menu, for files and folders: each streamed
    into the library, checked, the original deleted (its versions with it) and purged from the
    trash bin; a file that fails stays, versions and all, and the page says why. Nothing shared
    with the person and nothing in a team folder is moved. occ harvester:library-reset <user> --confirm: the admin deletes a person's library.
  • Save to Immich: a job's photos and videos uploaded into the person's Immich account
    (Immich's API of version 3), streamed, after Immich was asked for the content's SHA-1 (one
    there already is adopted, and still filed and tagged); an optional Album by a template
    (Magazines {year}), made once and reused; Immich's tags as Tag as (a/b nested); a
    photo deleted in Immich uploaded again; other files skipped for good with why; an unpacked
    ZIP's photos and videos alone. Your Immich account in the form: the server and the
    person's API key (a sensitive preference, tested as it is saved, never sent back;
    GET/PUT/DELETE /api/immich, POST /api/immich/test). The admin's Immich server
    (HARVESTER_IMMICH_URL) is preset for everyone and reached directly; a server a person types,
    through the guard's check; https:// unless Immich over plain http
    (HARVESTER_IMMICH_HTTP).
  • Save to a Cryptomator vault: its inbox (the folder gets the marker of the app Vaults
    for Cryptomator, which moves the files into the vault when the person unlocks it; files moved
    out are never fetched again) or the vault directly (each file encrypted on the server and
    written over WebDAV, harvester/vault.py; the password typed once, only the vault's keys kept,
    as a sensitive preference; a prominent warning that the server then holds the vault's keys).
    PUT /api/jobs/<id>/vault (the password), GET (whether the keys are kept), DELETE.
  • Tag as in a job's form: Nextcloud's tags (as before), Harvester's labels (in the app's
    database, shown and filtered in the job's Files and the library, never a Nextcloud tag)
    or none. GET /api/jobs/<id>/files: the files a job saved, with their tags or labels.
  • Browse beside the address of a gallery, video, feed or release job: the recorder's browser
    alone (nothing recorded), at the field's address, and Use this address takes the page it
    shows into the field.
  • Try the address in the form of every job but a website's (Try the mailbox, Try the
    remote): the listing once as the form holds it, before it is saved, nothing downloaded
    (POST /api/try): what a run would fetch (Files per run), what the job's history has, what
    its file types, sizes and patterns leave out, the first 20 files with the names and tags they
    would get, a mailbox's messages with what each gives, or why it failed with a hint. In a
    runner's place, through the guard, the proxy and the DNS servers, 60 seconds at most, one try
    of a person's at a time.
  • DNS servers of the admin's choice: DNS servers (a form of its own in Harvester's admin
    settings, or HARVESTER_DNS) sends every name the container looks up (the app and its guard,
    the browsers, the downloads, yt-dlp, gallery-dl, rclone, the mail servers) to plain addresses,
    DNS over TLS (tls://host[:port]) or DNS over HTTPS (https://host[:port]/path), a ClientID in
    the name or path, e.g. a filtering AdGuard Home. AdGuard's dnsproxy (in the image, pinned
    and checked) runs on 127.0.0.1:53 and /etc/resolv.conf names it; Nextcloud's, HaRP's and
    the proxy's names stay Docker's. Servers that do not answer fail the lookups, unless Docker's
    DNS when the servers fail (HARVESTER_DNS_FALLBACK, off by default) is on. The form's
    description is the status: whether the proxy runs and a test lookup works.
  • A name the DNS servers block (they answer 0.0.0.0 or ::) is told as such in the job's log,
    by the browser's guard and the downloads' checks (blocked by the DNS server), and a name they
    do not know as possibly blocked.
  • Fetch first in a mail job's form: Attachments, Linked files or Both. The form shows
    only the chosen part's options (an option's shown_when; a group whose options are all hidden
    goes with them) and keeps the others' values; a Linked files job asks for a link address or
    link text and nothing of attachment names. A job saved before keeps what it fetched (its
    attachments, and its linked files when Also the files linked in the message was on), read so
    (Source.upgraded, also for GET /api/jobs/<id> and a recipe file of before), not
    rewritten. A choice's words in the form (an option's labels).
  • A mail job's folder the mailbox does not have: the run's error names the folders it has, the
    closest first, ten of them (The mailbox has no folder “Rechnungen”. It has:
    INBOX/Rechnungen, Archive, INBOX); Try the mailbox checks the folder first and offers the
    mailbox's folders to take into the field (folders in POST /api/try's answer).
  • A mail job's run whose new messages are none of the sender and subject says so as a warning
    naming the patterns (Folder INBOX: none of the 12 new messages is from “…” with a subject
    like “…”).
  • Under a mail job's Pages to the on-demand job, while you have no on-demand website job:
    how to make one (record one for the portal, Website, On demand).

  Changed

  • The recorder's patterns in plain words: Address must match, Text must match (a pick
    by its text) and File's address must match (optional) (a JSON listing), each with a line
    saying that a part written (?P<name>…) becomes {name} for file names and tags, with an
    example.

  • A mail job's Pages to the on-demand job says what it is for in one sentence.

  • The progress log names the address a run looks at (Looking for new files at https://…) and
    says how many of the listed files the job has already (37 listed files are known already: not
    downloaded again), and the run's outcome counts them (0 downloaded, 37 known).

  • A website job's progress log names each page of file links with what it matched (page 2
    (https://…): 12 of 40 links match, 3 new, 20 pages a run) and says why a listing ended: a page
    of known files only (Stop at known files), or the Pages limit with a next page still there
    (Read 10 pages (Pages): older pages are not read this run). A level's waiting for … to show
    is said once a run, not for each of its listings.

  • Feed, video, file store and release jobs say in the log what their listing found: The feed
    “Reports”: 20 entries, 18 file links, 30 entries listed (Entries per run: 30) and the live
    or upcoming ones skipped, rclone listed 12 files in share:Reports (or that Files took
    none), 5 releases looked at (2 drafts or prereleases left out), 7 files match.

  • A download's waits and retries are lines in the log: … asks to wait 60 s (429, 503, a
    wait pattern) and …: the connection failed, trying again in 10 s, not a silent pause.

  • A video job's log has yt-dlp's warnings (ten a run, each once), and yt-dlp's advice for its
    command line is said as the job's options: Use --cookies-from-browser or --cookies reads
    Set the job's account: Cookies, likewise its user name and password, its formats and
    impersonation.

  • A watched page's run outcome says what changed: The page changed: -Tea: 2 EUR +Tea: 2.50 EUR
    (one line, 100 characters at most; The page changed alone before).

  • A mail job's progress log says what the mailbox search found (Folder INBOX: 12 new messages,
    3 from the sender and with the subject asked for, or no new messages and how many were done
    with before) and names each message of the sender and subject, with what the job takes from it
    (1 attachment, 2 links) or why nothing (nothing to fetch: no attachment of the names, types
    and sizes asked for, no link the patterns take): up to 50 a run, the rest counted.

  • The app's routes in groups: Try the address, recipe files and templates and the rclone wizard
    in harvester/routes/ (FastAPI's APIRouter), the jobs and the recorder in main.py; the API
    is the same.

  Fixed

  • A long run (a first one of hundreds of files) no longer pushes its own start out of the
    progress log: its first 30 lines stay beside the job's last 500.
  • A video larger than the job's largest file that yt-dlp finds so only as it downloads is skipped
    for good as one known to be too large (yt-dlp saved no file for …: larger than the job's
    largest file), not failed as yt-dlp left 0 files in every run.
  • A release job whose forge answers with an error or cannot be reached fails with its host,
    answer and what was asked (github.com answered 502 for the releases of o/r), or the address
    it could not reach, not with httpx's text; the log says how many releases were looked at and
    how many files match.
  • A gallery job whose gallery-dl lists part of the gallery and then fails (a page of many) says
    so in a warning with gallery-dl's error, and the run no longer counts as a complete pass; the
    log says how many files gallery-dl listed.
  • A run whose listing breaks after files came (a next page that fails, a feed cut short) still
    fails, but its reason keeps what it did: 3 downloaded; then the listing failed: … (only the
    listing's error before).
  • A linked file whose site cannot be reached (a name that does not resolve, a failed connection,
    an error 5xx, a redirect to a private address) no longer fails a mail job's run when nothing
    else came: it is skipped this time with a line in the log (skipped this time in the run's
    outcome), tried again in the next runs and given up after the third.
  • A file's failure in the progress log (file: why) and a selector's repair (level 1: selector
    repaired (… → …)) are translated (German, Polish), as their neighbours were.
  • The usual reasons a file fails or is skipped (… answered 404, … is not of the job's file
    types, … is larger than the job's largest file, … gave only 12 bytes), a run's (and 3
    more) and a file store's configuration errors are translated (German, Polish).
  • The run's reasons the start page has none of the recipe's links and … asks for a person,
    and the log's line of a page that loads none of the types to capture, are translated (German,
    Polish), as their neighbours were.
  • A recorded click that closes its own window (a sign-in popup's Allow) no longer fails its
    step now and then: the window may close before the browser reports the click as done.

  Downloads

  • harvester-0.14.0.tar.gz
    0 downloads · 2026-11-04 17:02:23 +00:00 · 1.7 MiB
  • signature.txt
    0 downloads · 2026-11-04 17:02:23 +00:00 · 684 B

  ---

  • Source code (ZIP)
    0 downloads
  • Source code (TAR.GZ)
    0 downloads