• myhloli's avatar
    refactor(pdf_parse_union): integrate LayoutLMv3 for block orderingReplace the... · 1efebe42
    myhloli authored
    refactor(pdf_parse_union): integrate LayoutLMv3 for block orderingReplace the heuristic-based block ordering algorithm with LayoutLMv3 model predictions toimprove the accuracy of block ordering on PDF pages. Additionally, refactor the span
    handling during block filling to ensure spans are correctly assigned.
    
    - Introduce LayoutLMv3ForTokenClassification from 'hantian/layoutreader' to predict block
      order.
    - Implement span replacement strategy to use pymu spans for non-OCR content.
    - Enhance cleanup process to free GPU memory more effectively after model use.
    - Adjust block ordering logic to use median line index for text, title, and interline equation blocks.
    - Refactor page parsing core logic for better maintainability.
    
    BREAKING CHANGE: The integration of LayoutLMv3 changes the internal block handling and
    ordering mechanism, which may affect downstream systems relying on the previous
    implementation. Ensure to test thoroughly before deployment.
    1efebe42
Name
Last commit
Last update
.github Loading commit data...
demo Loading commit data...
docs Loading commit data...
magic_pdf Loading commit data...
projects Loading commit data...
signatures/version1 Loading commit data...
tests Loading commit data...
web_api Loading commit data...
.gitignore Loading commit data...
.pre-commit-config.yaml Loading commit data...
Dockerfile Loading commit data...
LICENSE.md Loading commit data...
MinerU_CLA.md Loading commit data...
README.md Loading commit data...
README.md.bak Loading commit data...
README_ja-JP.md Loading commit data...
README_zh-CN.md Loading commit data...
README_zh-CN.md.bak Loading commit data...
magic-pdf.template.json Loading commit data...
requirements-docker.txt Loading commit data...
requirements-qa.txt Loading commit data...
requirements.txt Loading commit data...
setup.py Loading commit data...
update_version.py Loading commit data...