1. 08 Oct, 2024 4 commits
  2. 06 Oct, 2024 2 commits
  3. 30 Sep, 2024 6 commits
  4. 29 Sep, 2024 2 commits
  5. 28 Sep, 2024 3 commits
    • myhloli's avatar
      refactor(magic_pdf): import model helpers directly for clarity · 42a7d792
      myhloli authored
      Update import statements in `pdf_parse_union_core_v2.py` to directly import
      `prepare_inputs`, `boxes2inputs`, and `parse_logits` from `magic_pdf.model.v3.helpers`
      instead of from `magic_pdf.model.v3`. This change streamlines the imports, making the
      code more readable and maintaining a cleaner approach to modular design.
      42a7d792
    • myhloli's avatar
      refactor(pdf_parse_union_core_v2): update import paths to use new package structure · 5522d0a3
      myhloli authored
      Adapt import statements in `pdf_parse_union_core_v2.py` to reflect the updated packagestructure, changing from the `magic_pdf.v3.helpers` module to the `magic_pdf.model.v3`
      module. This ensures compatibility with the revised directory layout.
      5522d0a3
    • myhloli's avatar
      fix(pdf_parse): handle blocks without lines and enable bf16 on compatible devices · 2145a8b6
      myhloli authored
      Blocks without lines are now correctly indexed even when they contain textual content rendered
      as images. The sorting logic has been updated to accommodate this scenario. Additionally, the
      LayoutLMv3 model initialization has been enhanced to utilize bfloat16 precision on devices that
      support it, offering potential performance benefits on supported hardware.
      2145a8b6
  6. 27 Sep, 2024 23 commits
    • myhloli's avatar
      refactor(pdf_parse): remove redundant sorting and optimize block indexing · 177ab08e
      myhloli authored
      Removed redundant sorting of lines by model and optimized calculation of block
      indexes by using a single pass through the sorted lines. This change simplifies the
      code and potentially improves performance by reducing the number of sortingoperations and unnecessary iterations over blocks without lines.
      177ab08e
    • myhloli's avatar
      refactor(draw_bbox): remove commented-out code and streamline bbox... · 83c07387
      myhloli authored
      refactor(draw_bbox): remove commented-out code and streamline bbox drawingRemoved legacy commented-out code related to layout_bbox_list from draw_bbox.py, which
      was used for diagnostic purposes and was no longer necessary. This change streamlines
      the codebase and clarifies the drawing process of bounding boxes on PDF pages. The update
      also adjusts the order of operations slightly for improved readability without altering
      the functionality.
      83c07387
    • myhloli's avatar
      feat(requirements): add torch and transformers libraries · 65615455
      myhloli authored
      Introduce torch and transformers libraries to support new ML features.Ensure version compatibility by adding torch version within the range 2.2.2 to 2.3.1and include the necessary transformers library.
      65615455
    • myhloli's avatar
      refactor(pdf_parse_union_core_v2): implement model initialization within... · b9dfdea3
      myhloli authored
      refactor(pdf_parse_union_core_v2): implement model initialization within classRefactored model initialization to be handled by a singleton class to ensure that model
      instances are reused across calls, avoiding redundant initializations. Removed logger
      information that was commented out and ensured consistency in logging behavior.
      b9dfdea3
    • myhloli's avatar
      refactor(drawing): simplify draw bbox functions and adjust debug config · b2790f6f
      myhloli authored
      Refactor the draw bbox functions by removing unused imports and simplifying the
      code logic for drawing layout and line sorting bounding boxes. Adjust the debug
      configuration to enable content list dumping and disable markdown making mode.
      b2790f6f
    • myhloli's avatar
      Merge remote-tracking branch 'origin/add-layoutreader' into add-layoutreader · 16b51c79
      myhloli authored
      # Conflicts:
      #	magic_pdf/libs/draw_bbox.py
      16b51c79
    • myhloli's avatar
      feat(draw_bbox): add option to toggle bounding box drawing · 43a57d56
      myhloli authored
      Introduce an additional argument `draw_bbox` in the `draw_bbox_with_number` function to
      enable toggling the drawing of bounding boxes on or off. When set to `False`, no bounding
      box will be drawn, allowing for situations where only text
      43a57d56
    • myhloli's avatar
      refactor(draw_bbox): remove conditional layout bbox drawing · c56de493
      myhloli authored
      Remove debug code related to layout bbox visualization and adjust drawing functions to
      support optional line sorting bboxes. This change includes the removal of `draw_layout_bbox`
      function and updates to `draw_bbox_with_number` to support variable line width for bbox drawing.
      c56de493
    • myhloli's avatar
      refactor(draw_bbox): add line sorting visualization · 34f89650
      myhloli authored
      Add a new function `draw_line_sort_bbox` to visualize the sorting of lines on each page.
      This includes indexing lines and handling both text and non-text elements such as tables
      and images for better content organization.
      
      Also, comment out GPU-related code for flexibility and remove overlaps in bounding box
      detection, which improves the accuracy of layout splitting.
      34f89650
    • myhloli's avatar
      refactor(pdf_parse_union): integrate LayoutLMv3 for block orderingReplace the... · 1efebe42
      myhloli authored
      refactor(pdf_parse_union): integrate LayoutLMv3 for block orderingReplace the heuristic-based block ordering algorithm with LayoutLMv3 model predictions toimprove the accuracy of block ordering on PDF pages. Additionally, refactor the span
      handling during block filling to ensure spans are correctly assigned.
      
      - Introduce LayoutLMv3ForTokenClassification from 'hantian/layoutreader' to predict block
        order.
      - Implement span replacement strategy to use pymu spans for non-OCR content.
      - Enhance cleanup process to free GPU memory more effectively after model use.
      - Adjust block ordering logic to use median line index for text, title, and interline equation blocks.
      - Refactor page parsing core logic for better maintainability.
      
      BREAKING CHANGE: The integration of LayoutLMv3 changes the internal block handling and
      ordering mechanism, which may affect downstream systems relying on the previous
      implementation. Ensure to test thoroughly before deployment.
      1efebe42
    • sfk's avatar
      Update README_zh-CN.md · 9c38c880
      sfk authored
      9c38c880
    • Xiaomeng Zhao's avatar
      Merge pull request #670 from myhloli/dev · 0b5d8881
      Xiaomeng Zhao authored
      docs: update project lists in README files to include web_api
      0b5d8881
    • myhloli's avatar
      docs: update project lists in README files to include web_api · 39873969
      myhloli authored
      Add the web_api project to the lists of projects in both the English and Chinese
      README.md files, providing a brief description and linking to the project's
      documentation. Ensure that the formatting and style are consistent with the
      existing project entries.
      39873969
    • myhloli's avatar
      refactor(draw_bbox): clear cuda cache and update bbox sorting · 36220d69
      myhloli authored
      - Added CUDA cache clearing after layoutreader prediction to free up GPU memory.
      - Modified the bbox sorting logic to sort text and title blocks separately.
      - Adjusted drawing colors for better distinction in debug visualizations.
      36220d69
    • Xiaomeng Zhao's avatar
      Merge pull request #669 from LollipopsAndWine/dev · 26e36262
      Xiaomeng Zhao authored
      feat: 删除无用的文件,更新前端style
      26e36262
    • decrystal's avatar
      Merge pull request #9 from LollipopsAndWine/feat/ade/dev · 9b88e2e3
      decrystal authored
      feat: style
      9b88e2e3
    • dechen lin's avatar
      feat: style · 2bd254d0
      dechen lin authored
      2bd254d0
    • sfk's avatar
      Update README.md · 684e8705
      sfk authored
      update backlog
      684e8705
    • sfk's avatar
      Update README_zh-CN.md · 0aa1a983
      sfk authored
      update backlog
      0aa1a983
    • Xiaomeng Zhao's avatar
      Delete README_zh-CN.md.bak · 52c95bba
      Xiaomeng Zhao authored
      52c95bba
    • Xiaomeng Zhao's avatar
      Delete README.md.bak · ab92f455
      Xiaomeng Zhao authored
      ab92f455
    • sfk's avatar
      Update README.md · 7da752eb
      sfk authored
      update backlog
      7da752eb
    • Xiaomeng Zhao's avatar
      Update README.md · ac869888
      Xiaomeng Zhao authored
      ac869888