Zio.ingestion()
The ingestion pipeline directly handles the original data (ingestion_data table) that it fills. Anything that comes in via email, database, spreadsheet, web, or API is stored here as raw_data. Querying, counting, creating, updating, and deleting (CRUD) are performed in the same way as with Zio.entity.
Don’t Guess Field Names
Section titled “Don’t Guess Field Names”The ingestion schema varies completely by instance and industry (food, architecture, fashion, …). First check which schemas exist and what fields they contain using
Zio.ingestion.schemas(). If you guess field names, the result will quietly be empty.
What can I ask
Section titled “What can I ask”Below <스키마명> · <필드명> are placeholders. Use the actual names obtained from schemas().
| When you receive this request | Then do this |
|---|---|
| What is being collected? / What fields exist? | Zio.ingestion.schemas() |
| Show me the most recent data for OO | Zio.ingestion("<스키마명>").order_by("-ingested_at").fetch() |
| Show only those that match this condition | .where(**{"<필드명>": "<값>"}) .fetch() |
| How many match this condition? | .where(**{"<필드명>": "<값>"}) .count() |
| Match any of several fields (OR) | .where_or(status__contains="x", sender_email__contains="y").fetch() groups multiple conditions on the same schema with OR (different from other .where() which are AND). Even if the top columns and raw_data field are mixed within a single group, it works. Calling it several times creates separate OR groups. The suffix is the same as .where(). It applies only to read (fetch, count) operations — applying OR to update or delete can affect more records than intended, so it is not allowed. |
| Only those that are unprocessed / errored | .where(status="error").fetch() · process_state bit condition |
| Fix a value / change a status | .where(...).update({...}) |
| Delete these | .where(...).delete() |
| Manually insert one record | Not allowed — records can only enter through the collection pipeline. See .create(props) below |
Relationships and group aggregations are not here
Section titled “Relationships and group aggregations are not here”Questions like “who traded with whom” (relationship queries) or “how many per item value” (group by aggregation) must be looked at after analysis in the knowledge graph →
Zio.node. Here (Zio.ingestion) is the original data, so it only supports conditional querying and.count().
Collection Data Fluent CRUD API
Section titled “Collection Data Fluent CRUD API”| Method | Description |
|---|---|
Zio.ingestion.schemas() |
Retrieves a list of collection schemas. Each item is {name, group, description, count, columns[]} — name is the unique identifier (used as an address), group is the pipeline type (e.g., odbc, shared among many), and columns contains the actual field names and types of that schema. Call this before querying. |
Zio.ingestion(schema) |
The starting point for Fluent chaining on a single collection schema. Pass the name returned by schemas() in schema — name is uniquely enforced, so it becomes the official address. If you provide a group name, you receive the list of schema names within that group instead of an address; choose one from the list. (The numeric schema_id can also be supplied, but with unique names it’s unnecessary.) |
.where(**filters) |
Applies conditions. Both fields inside raw_data and top‑level columns (id, status, process_state, source_identifier, ingested_at) are allowed. Suffixes include: __contains, __gte, __lte, __in, etc. If a field name contains Korean or spaces, pass it as .where(**{"필드명": value}). |
.order_by(*fields) |
Sorting. Prefixing a field with - sorts descending (e.g., "-ingested_at"). |
.fetch(limit=100, skip=0, include_derived=True) |
Retrieves data matching the conditions as a GraphResult. Each record is complete — schema‑specific fields + common metadata + common derived (security_scan, ocr, inference_result, analysis_result). See the “Structure of One Record” below. If large queries burden derived calculations, set include_derived=False (original data + metadata only, faster). |
.count() |
Counts records that match the conditions. (No aggregation by groups — see above.) |
.create(props) |
Cannot be used. Records are inserted only through the collection pipeline—duplicate checks (PK), structuring, and status handling are enforced by the pipeline. If called, it will explain why and reject. New data is ingested via the schema’s pipeline (collection connection + schedule). For past dates, call something like @getDate(-7). To re‑fetch a corrected past record, delete that record and let the pipeline fetch it again—existing PKs are skipped by design. |
.where(...).update(props) |
Updates rows that match the conditions. You can change raw_data fields as well as status and process_state. If no condition (where) is supplied, the operation is rejected. |
.where(...).delete() |
Deletes rows that match the conditions, along with related items (analysis, deep analysis, attachments, inference queue) and attachment files on disk. Irreversible; if no condition is supplied, it is rejected. |
.attachments(id) |
Retrieves the list of attachments for a collection record (id = _ingestion_id) — metadata + OCR text. Each item is {file_name, file_size, saved, cdr_status, ocr_status, ocr_text, ocr_meta}. saved=False indicates blocked or non‑stored body‑check attachment (no stored file). Attachments are only accessible in the context of that collection record. |
.attachment(id, "파일명") |
Handles a single attachment. .bytes() returns sanitized body bytes; .text() returns OCR extracted text (for image attachments, None if absent); .save(["relative_path"]) saves the file to the skill folder (if no path is given, uses the original filename; only relative paths are allowed—outside the folder is prohibited). |
Structure of One Record — what fetch() returns
Section titled “Structure of One Record — what fetch() returns”A record consists of three parts. ① Varies per schema, while ② and ③ look the same for any collection record — identical to the agent’s state["input"].
| Part | Content |
|---|---|
| ① Schema‑specific (variable) | Flattened fields from raw_data — e.g., subject, body, sender_email, … Do not guess field names; verify them via schemas()’s columns. |
| ② Common metadata (fixed) | _ingestion_id, id, status, process_state, source_identifier, ingested_at, schema_id |
| ③ Common derived (fixed structure) | Four sections filled by the engine and analysis. Even if empty, the key structure remains (not an empty {}): security_scan, ocr, inference_result, analysis_result. |
# fetch() 한 건의 완전한 모양 — ② 공통 메타 · ③ 공통 파생은 늘 이 구조{ # ① 스키마별 (schemas() 의 columns 로 확인 — 아래는 예시일 뿐, 실제 필드는 다름) "subject": "...", "body": "...", "sender_email": "...",
# ② 공통 메타 "_ingestion_id": 20543, "id": 20543, "status": "done", "process_state": 63, "source_identifier": "<...>", "ingested_at": "2026-11-11T03:14:41-05:00", "schema_id": 1,
# ③ 공통 파생 — 고정 구조 (비어도 키는 그대로) "analysis_result": { }, # 1차 정적 분석 (키는 설계별) "inference_result": { # 2차 AI 최종 추론 "spam_yn": true, "verdict": "...", "reason": "...", "scanner_engine": "...", "detected_indicators": [] }, "security_scan": { # 정밀검사 (보안 엔진) "status": "safe", "av_infected": false, "yara_matched": false, "url": { "total": 0, "malicious": 0, "suspicious": [] }, "cdr_cleaned": 0 }, "ocr": { # OCR 추출 (이미지 글자) "extracted": false, "status": "no_image", "count": 0, "char_count": 0, "items": [], "merged_text": "" }}Why is
Section titled “Why is ocr.status “0건”?”ocr.status“0건”?The status distinguishes between three cases:
ok(image with extracted text,count > 0),no_text(image processed but zero characters),no_image(no image attachment for OCR), andengine_off(OCR engine is off—the operator must enable it via/monitoring). Each case is identified by this single field. Each item inocr.items[]is{file_name, text, char_count, avg_conf};ocr.merged_textconcatenates the extracted text from each attachment with a header per file for consumption.
Attachment Files — .attachments(id) · .attachment(id, "파일명")
Section titled “Attachment Files — .attachments(id) · .attachment(id, "파일명")”Treats sanitized (CDR) attachments linked to a single collection record as part of that record’s context. It does not open the entire filesystem; it scopes by (schema · collection ID). If you only need text, you usually don’t need to reach this level — fetch()’s ocr.merged_text already contains extracted text from attachments. Use the following only when directly handling the original file (e.g., a PDF or Excel listed in the table).
SCHEMA = "<스키마명>" # schemas() 의 nameING = 844 # 수집 건 id (= fetch 의 _ingestion_id)
# ① 이 건에 뭐가 붙었나 — 맥락(메타 + OCR 글자)for a in Zio.ingestion(SCHEMA).attachments(ING): print(a["file_name"], a["file_size"], a["ocr_status"])
f = Zio.ingestion(SCHEMA).attachment(ING, "invoice.pdf")raw = f.bytes() # 소독된 본문 바이트 (그대로 파싱)text = f.text() # OCR 글자(이미지 첨부) · 없으면 Nonepath = f.save() # 스킬 폴더에 'invoice.pdf' 로 저장path = f.save("inbox/a.pdf") # 하위 경로(상대만) — 웹 탐색기 · SSH 서 그대로 보임Only sanitized files are downloaded (blocked or unsaved attachments have
saved=False, and.bytes()returns 404)..save()cannot write outside the skill folder (only relative paths). The instance is read‑only and only touches the collection record of this instance.
Process History & Status — process_state and status
Section titled “Process History & Status — process_state and status”The extent to which data has progressed through processing is indicated by process_state (bitwise accumulation), while whether an engine will handle it is determined by status. Changing these values via update triggers a cron worker to re‑process automatically—re‑execution itself is not performed by this SDK.
| process_state | Integer value | Meaning |
|---|---|---|
| Bit 0 | 1 |
Collection complete |
| Bit 1 | 2 |
Data enrichment (AI missing data correction) |
| Bit 2 | 4 |
Category classification |
| Bit 3 | 8 |
Aggregation / summarization |
| Bit 4 | 16 |
Agent (Dynamic) · Knowledge graph processing |
| Bit 5 | 32 |
Final inference |
| status | Engine | Meaning |
|---|---|---|
pending |
Running | Awaiting processing (new or manually triggered re‑processing) |
processing |
Locked | Worker in progress (deduplication protection) |
done |
Stopped | Current assigned process finished |
error |
Stopped | Halted due to exception (needs review) |
paused |
Stopped | Manually paused by user |
Resetting the collection stage (Bits 0–3) automatically resets subsequent bits (Bits 4–5) to 0, preserving freshness.
Example (actual schema and field names should be verified via schemas() — placeholders below)
Section titled “Example (actual schema and field names should be verified via schemas() — placeholders below)”from zio_ontology import Zio
# 1. 무엇을 수집하고 있나 — 필드명은 여기서 확인한다 (인스턴스마다 다르다)for s in Zio.ingestion.schemas(): print(s["name"], s["count"], [c["name"] for c in s["columns"]])
# 2. 조건에 맞는 건수 세기# <스키마명> · <필드명> 은 위에서 확인한 실제 이름으로 바꾼다n = Zio.ingestion("<스키마명>").where(**{"<필드명>": "<값>"}).count()
# 3. 최근 것부터 100건 조회rows = Zio.ingestion("<스키마명>").order_by("-ingested_at").fetch(limit=100).data
# 4. 조건에 맞는 값 고치기 (where 없으면 거절)Zio.ingestion("<스키마명>").where(**{"<필드명>": "<값>"}).update({"<필드명>": "<새값>"})
# 5. 상태를 바꿔 재처리 흐르게 하기 (cron 이 알아서 다시 처리)Zio.ingestion("<스키마명>").where(id=123).update({"status": "pending", "process_state": 1})
# 6. 조건에 맞는 것 지우기 (딸린 것·첨부 파일까지 함께, 되돌릴 수 없음)Zio.ingestion("<스키마명>").where(**{"<필드명>": "<값>"}).delete()Modifying the original does not update derived results
Section titled “Modifying the original does not update derived results”Updating
raw_datawithupdatedoes not automatically change already created knowledge graph or analysis results. To reflect changes in analysis results, either reset the relevant bit(s) ofprocess_state(e.g., keep only the collection bit and set others to 0) so that cron re‑processes, or directly modify the graph viaZio.node.
(주)지오유 · 경기 용인시 처인구 유림로154번길(유방동 673-2) 20-13
CEO 신달수·Business no. 142-81-36612·E-commerce no. 용인처인-0118호·Privacy officer 신달수·Email shindalsoo@gmail.com·Tel 1833-3623·Fax 02-6918-4006