genroc

    docs / Guides / Operating

    Large values

    How genroc manages large values in process instances.

    Genroc stores large values separately and doesn't return them directly when you communicate with the API.

    What is a large value?

    Genroc measures each stored value on its own. When one is over its limit, genroc moves its largest parts into objects until the rest fits.

    Stored valueWhere you see itLimit
    Process inputstate.input2 KiB
    Each task's output, separatelystate.outputs.<task id>2 KiB
    Process outputoutput2 KiB
    The last errorstate.last_error2 KiB
    A raised error's dataerror_data2 KiB
    External task inputexternal_input2 KiB
    A log entry's payloadthe entry's data2 KiB

    Ten task outputs of 1.5 KB each are therefore all kept inline: each one is under its own limit.

    A log entry's payload over --log-payload-max (default 1 MiB) is not stored at all. The entry's meta.payload_omitted_bytes says how big it was.

    What it looks like

    Say a task downloads a web page and keeps the HTML (about 180 KB) in its output. The instance detail doesn't contain the HTML. The body field is null, and the objects section says where it was and how to get it:

    {
      "id": "5fy765kh",
      "status": "completed",
      "state": {
        "outputs": {
          "download": {
            "status": 200,
            "content_type": "text/html",
            "body": null
          }
        }
      },
      "objects": [
        {
          "path": ["state", "outputs", "download", "body"],
          "ref": "9f2ac1b4e7d05f38a61c0d2e4b7f9a13",
          "size": 184320
        }
      ]
    }

    path points to the null field, ref is the id of the stored value and size is its size in bytes. Only the large parts are moved out: status and content_type stay where they are.

    To get the value, use the ref:

    > curl http://localhost:8448/api/objects/9f2ac1b4e7d05f38a61c0d2e4b7f9a13
    {"data": "\"<!doctype html><html>...\""}

    data is the value encoded as JSON text, so parse it once more to get the original value.

    The CLI does this for you: genctl detail shows {ref, size} in place of the null, genctl detail --resolve fills it in and genctl object <ref> prints one value.

    Which endpoints

    EndpointWhat can be moved outPaths start at
    GET /api/instances/{id}output, error_data, external_inputthe response
    GET /api/instances/{id}/detailanything in statethe response
    GET /api/instances/{id}/logseach entry's datathe log entry
    POST /api/external-tasks/claimeach item's external_inputthe item

    No endpoint fills the values in for you; fetch each one with GET /api/objects/{ref}.

    Fetching an object needs the read or worker permission.