# RAG Ingestion

Request `formats=json,markdown`, chunk the Markdown by document structure, and attach JSON metadata so retrieved passages can point back to a page and a bounding box.

Source: https://docs.docushell.com/rag-ingestion
Category: Guide

## Related

- [Parse PDF](/parse-pdf.md): The endpoint, its artifacts, and the extraction controls used here.
- [Ethos Verification](/ethos.md): Verify that cited evidence actually binds to the source before releasing an answer.
- [Try Parse live](/playgrounds/parse): Inspect overlays, extracted blocks, and JSON before wiring a pipeline.

## Chunk By Structure, Not By Character Count

Blind character windows split headings from their paragraphs and rows from their tables. Start from document semantics instead: headings, paragraphs, lists, captions, and tables. Merge short neighboring elements, keep tables intact, and carry page numbers and bounding boxes into vector metadata.

The JSON artifact is also the audit layer. Store enough metadata to reproduce a citation, highlight a source region in a review UI, and debug bad retrieval without reparsing the PDF.

- Request `formats=json,markdown`. Chunk and embed the Markdown; attach metadata from the JSON.
- Start new chunks at major headings and keep heading-plus-paragraph groups together.
- Keep table nodes as standalone chunks when row and column relationships carry the meaning.
- Store source file ID, page number, heading path, node type, and bounding box per chunk.
- Leave rendering-mismatch defenses on for untrusted PDFs so hidden and off-page text stays out of model context.
- Leave header/footer exclusion on unless repeated page furniture is part of the answer.

> Cite the DocuShell metadata, not just the text string you sent to the model. Page and bounding-box metadata make citations inspectable.

## Request And Metadata Shape

### Submit a RAG-ready parse job

```bash
curl -X POST "https://api.docushell.com/api/v1/parse" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Idempotency-Key: rag-parse-001" \
  -F "file=@./policy-handbook.pdf;type=application/pdf" \
  -F "formats=json,markdown" \
  -F "reading_order=xycut" \
  -F "use_struct_tree=true"
```

### Chunk metadata shape

```json
{
  "content": "## Data Retention\n\nCustomer documents are retained for the configured retention window...",
  "metadata": {
    "source_file_id": "file_01JX...",
    "source_name": "policy-handbook.pdf",
    "section": "Data Retention",
    "node_types": ["heading", "paragraph"],
    "page_start": 3,
    "page_end": 4,
    "bounding_boxes": [
      { "page": 3, "bbox": { "x": 0.88, "y": 1.21, "w": 6.21, "h": 0.52 } }
    ]
  }
}
```

Use whichever metadata keys your system prefers, but preserve the DocuShell page and bounding-box data.

## Tagged PDFs And Structure Trees

When a source PDF carries usable structure tags, `use_struct_tree=true` tells DocuShell to prefer them for reading order and hierarchy. That improves headings, lists, table relationships, and chunk boundaries.

Real collections are mixed: some PDFs are well tagged, some untagged, some tagged badly. Compare both settings on representative files during integration rather than committing to one strategy blind.

- Use `use_struct_tree=true` for tagged policy documents, manuals, reports, and accessible PDFs.
- Use the default layout-aware path for untagged or poorly tagged files.
- Request `formats=tagged_pdf` when you need a generated tagged artifact for accessibility review.

## Validate Before You Index

Three views of the same parse job, all in the Parse Playground.

- [Annotated PDF viewer](/playgrounds/parse): Source pages overlaid with layout boxes, category tags, and reading-order numbers.
- [Blocks and tables](/playgrounds/parse): Every extracted node as data: order, category, page, bounding box, and content.
- [JSON, Markdown, and text](/playgrounds/parse): Compare the structured JSON against its text companions from one job.