Merge branch 'master' of github.com:papayalove/Magic-PDF

# Conflicts: # docs/how_to_download_models_zh_cn.md

Merge branch 'master' of github.com:papayalove/Magic-PDF
# Conflicts: # docs/how_to_download_models_zh_cn.md
b8adb630 · liukaiwen · 6de68f06 · 52069612 · b8adb630 · b8adb630
Commit b8adb630 authored Aug 05, 2024 by liukaiwen
42 changed files
--- a/.gitignore
+++ b/.gitignore
@@ -35,3 +35,5 @@ ocr_demo

 /app/common/__init__.py
 /magic_pdf/config/__init__.py
+source.dev.env
+
--- a/README_zh-CN.md
+++ b/README_zh-CN.md
@@ -121,7 +121,9 @@ pip install magic-pdf[full]==0.6.2b1 -i https://pypi.tuna.tsinghua.edu.cn/simple
 #### 2. 下载模型权重文件

 详细参考 [如何下载模型文件](docs/how_to_download_models_zh_cn.md)  
-下载后请将models目录移动到空间较大的ssd磁盘目录  
+> ❗️模型下载后请务必检查模型文件是否下载完整
+> 
+> 请检查目录下的模型文件大小与网页上描述是否一致，如果可以的话，最好通过sha256校验模型是否下载完整

 #### 3. 拷贝配置文件并进行配置
 在仓库根目录可以获得 [magic-pdf.template.json](magic-pdf.template.json) 配置模版文件
@@ -133,9 +135,11 @@ cp magic-pdf.template.json ~/magic-pdf.json
 ```

 在用户目录中找到magic-pdf.json文件并配置"models-dir"为[2. 下载模型权重文件](#2-下载模型权重文件)中下载的模型权重文件所在目录
-> ❗️务必正确配置模型权重文件所在目录，否则会因为找不到模型文件而导致程序无法运行
+> ❗️务必正确配置模型权重文件所在目录的【绝对路径】，否则会因为找不到模型文件而导致程序无法运行
+> 
+> windows系统中此路径应包含盘符，且需把路径中所有的"\"替换为"/",否则会因为转义原因导致json文件语法错误。
 > 
-> windows系统中应把路径中所有的"\\"替换为"/",否则会因为转义原因导致json文件语法错误。
+> 例如：模型放在D盘根目录的models目录，则model-dir的值应为"D:/models"
 ```json
 {
  "models-dir": "/tmp/models"

--- a/README_zh-CN_v2.md
+++ b/README_zh-CN_v2.md
@@ -16,12 +16,11 @@
 <a href="https://trendshift.io/repositories/11174" target="_blank"><img src="https://trendshift.io/api/badge/repositories/11174" alt="opendatalab%2FMinerU | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/></a>

 <!-- language -->
-[English](README.md) | [简体中文](README_zh-CN.md) | [日本語](README_ja-JP.md)
+[English](README.md) | [简体中文](README_zh-CN.md)


 <!-- hot link -->
 <p align="center">
-<a href="https://github.com/opendatalab/MinerU">MinerU: 端到端的PDF解析工具（基于PDF-Extract-Kit）支持PDF转Markdown</a>🚀🚀🚀<br>
 <a href="https://github.com/opendatalab/PDF-Extract-Kit">PDF-Extract-Kit: 高质量PDF解析工具箱</a>🔥🔥🔥
 </p>

@@ -35,7 +34,8 @@

 # 更新记录

- 2024/07/18 首次开源
+- 2024/08/01 0.6.2b1发布，优化了依赖冲突问题和安装文档
+- 2024/07/05 首次开源


 <!-- TABLE OF CONTENT -->
@@ -82,7 +82,10 @@
 # MinerU
 ## 项目简介
 MinerU是一款将PDF转化为机器可读格式的工具（如markdown、json），可以很方便地抽取为任意格式。
-MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练过程中，我们将会集中精力解决科技文献中的符号转化问题，以此在大模型时代为科技发展做出一点贡献。
+MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练过程中，我们将会集中精力解决科技文献中的符号转化问题，希望在大模型时代为科技发展做出贡献。
+相比国内外知名商用产品MinerU还很年轻，如果遇到问题或者结果不及预期请到issue提交问题，同时附上相关PDF。
+
+https://github.com/user-attachments/assets/4bea02c9-6d54-4cd6-97ed-dff14340982c

 ## 主要功能

@@ -102,12 +105,12 @@ MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练
 如果遇到任何安装问题，请先查询 <a href="#faq">FAQ</a> </br>
 如果遇到解析效果不及预期，参考 <a href="#known-issue">Known Issue</a></br>
 有3种不同方式可以体验MinerU的效果：
- 在线体验(无需任何安装)
- 使用CPU快速体验（Windows，Linux，Mac）
- Linux/Windows + CUDA
+- [在线体验(无需任何安装)](#在线体验)
+- [使用CPU快速体验（Windows，Linux，Mac）](#使用cpu快速体验)
+- [Linux/Windows + CUDA](#使用gpu)


-**软硬件环境支持说明**
+**⚠️安装前必看——软硬件环境支持说明**

 为了确保项目的稳定性和可靠性，我们在开发过程中仅对特定的软硬件环境进行优化和测试。这样当用户在推荐的系统配置上部署和运行项目时，能够获得最佳的性能表现和最少的兼容性问题。

@@ -166,23 +169,47 @@ MinerU诞生于[书生-浦语](https://github.com/InternLM/InternLM)的预训练

 ### 在线体验

-[在线体验点击这里](TODO)
+[在线体验点击这里](https://opendatalab.com/OpenSourceTools/Extractor/PDF)


 ### 使用CPU快速体验

+#### 1. 安装magic-pdf

 ```bash
 pip install magic-pdf[full]==0.6.2b1 detectron2 --extra-index-url https://wheels.myhloli.com -i https://pypi.tuna.tsinghua.edu.cn/simple
 ```
+#### 2. 下载模型权重文件
+
+详细参考 [如何下载模型文件](docs/how_to_download_models_zh_cn.md)
+> ❗️模型下载后请务必检查模型文件是否下载完整
+> 
+> 请检查目录下的模型文件大小与网页上描述是否一致，如果可以的话，最好通过sha256校验模型是否下载完整
+
+#### 3. 拷贝配置文件并进行配置
+在仓库根目录可以获得 [magic-pdf.template.json](magic-pdf.template.json) 配置模版文件
+> ❗️务必执行以下命令将配置文件拷贝到【用户目录】下，否则程序将无法运行
+> 
+>  windows的用户目录为 "C:\Users\用户名", linux用户目录为 "/home/用户名", macOS用户目录为 "/Users/用户名"
+```bash
+cp magic-pdf.template.json ~/magic-pdf.json
+```

-> ❗️已收到多起由于镜像源和依赖冲突问题导致安装了错误版本软件包的反馈，请务必安装完成后通过以下命令验证版本是否正确
-> ```bash
-> magic-pdf --version
-> ```
-> 如版本低于0.6.2b1，请提交issue进行反馈。
+在用户目录中找到magic-pdf.json文件并配置"models-dir"为[2. 下载模型权重文件](#2-下载模型权重文件)中下载的模型权重文件所在目录
+> ❗️务必正确配置模型权重文件所在目录的【绝对路径】，否则会因为找不到模型文件而导致程序无法运行
+>
+> windows系统中此路径应包含盘符，且需把路径中所有的"\"替换为"/",否则会因为转义原因导致json文件语法错误。
+> 
+> 例如：模型放在D盘根目录的models目录，则model-dir的值应为"D:/models"
+```json
+{
+  "models-dir": "/tmp/models"
+}
+```

 ### 使用GPU
+如果您的设备支持CUDA，且满足主线环境中的显卡要求，则可以使用GPU加速，请根据自己的系统选择适合的教程：
+
 - [Ubuntu22.04LTS + GPU](docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md)
 - [Windows10/11 + GPU](docs/README_Windows_CUDA_Acceleration_zh_CN.md)

@@ -191,7 +218,45 @@ pip install magic-pdf[full]==0.6.2b1 detectron2 --extra-index-url https://wheels

 ### 命令行

-TODO
+```bash
+magic-pdf --help
+Usage: magic-pdf [OPTIONS]
+
+Options:
+  -v, --version                display the version and exit
+  -p, --path PATH              local pdf filepath or directory  [required]
+  -o, --output-dir TEXT        output local directory
+  -m, --method [ocr|txt|auto]  the method for parsing pdf.  
+                               ocr: using ocr technique to extract information from pdf,
+                               txt: suitable for the text-based pdf only and outperform ocr,
+                               auto: automatically choose the best method for parsing pdf
+                                  from ocr and txt.
+                               without method specified, auto will be used by default. 
+  --help                       Show this message and exit.
+
+
+## show version
+magic-pdf -v
+
+## command line example
+magic-pdf -p {some_pdf} -o {some_output_dir} -m auto
+```
+
+其中 `{some_pdf}` 可以是单个pdf文件，也可以是一个包含多个pdf文件的目录。
+运行完命令后输出的结果会保存在`{some_output_dir}`目录下, 输出的文件列表如下
+
+```text
+├── some_pdf.md                 # markdown 文件
+├── images                      # 存放图片目录
+├── layout.pdf                  # layout 绘图
+├── middle.json                 # minerU 中间处理结果
+├── model.json                  # 模型推理结果
+├── origin.pdf                  # 原 pdf 文件
+└── spans.pdf                   # 最小粒度的bbox位置信息绘图
+```
+
+更多有关输出文件的信息，请参考[输出文件说明](docs/output_file_zh_cn.md)
+

 ### API

@@ -245,6 +310,8 @@ TODO
 - 阅读顺序基于规则的分割，在一些情况下会乱序
 - 列表、代码块、目录在layout模型里还没有支持
 - 漫画书、艺术图册、小学教材、习题尚不能很好解析
+- 在一些公式密集的PDF上强制启用OCR效果会更好
+- 如果您要处理包含大量公式的pdf,强烈建议开启OCR功能。使用pymuPDF提取文字的时候会出现文本行互相重叠的情况导致公式插入位置不准确。

 好消息是，这些我们正在努力实现！

@@ -267,6 +334,7 @@ The project currently leverages PyMuPDF to deliver advanced functionalities; how

 # Acknowledgments

+- [StructEqTable](https://github.com/UniModal4Reasoning/StructEqTable-Deploy)
 - [PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)
 - [PyMuPDF](https://github.com/pymupdf/PyMuPDF)
 - [fast-langdetect](https://github.com/LlmKira/fast-langdetect)

--- a/demo/small_ocr.pdf
+++ b/demo/small_ocr.pdf
--- a/docs/FAQ_zh_cn.md
+++ b/docs/FAQ_zh_cn.md
@@ -50,20 +50,7 @@ pip install paddlepaddle==3.0.0b1
 可能是由于模型文件未下载完整导致，可尝试重现下载模型文件后再试  
 参考：https://github.com/opendatalab/MinerU/issues/143

-### 7.程序运行完成后，找不到tmp目录
-
-程序输出目录是在"magic-pdf.json"中通过
-```json
-{
-  "temp-output-dir": "/tmp"
-}
-```
-进行配置的。  
-如果没有更改这个参数，使用默认的配置执行程序，在linux/macOS会在绝对路径"/tmp"下创建一个"magic-pdf"文件夹作为输出路径。
-而在windows下，默认的输出路径与执行命令时，命令行所在的盘符相关，如果命令行在C盘，则默认输出路径为"C://tmp/magic-pdf"。  
-参考：https://github.com/opendatalab/MinerU/issues/149
-
-### 8.模型文件应该下载到哪里/models-dir的配置应该怎么填
+### 7.模型文件应该下载到哪里/models-dir的配置应该怎么填

 模型文件的路径输入是在"magic-pdf.json"中通过
 ```json
@@ -75,24 +62,16 @@ pip install paddlepaddle==3.0.0b1
 这个路径是绝对路径而不是相对路径，绝对路径的获取可在models目录中通过命令 "pwd" 获取。  
 参考：https://github.com/opendatalab/MinerU/issues/155#issuecomment-2230216874

-### 9.命令行中 --model "model_json_path" 指的是什么？
+### 8.命令行中 --model "model_json_path" 指的是什么？

 model_json 指的是通过模型分析后生成的一种有特定格式的json文件。  
 如果使用 https://github.com/opendatalab/PDF-Extract-Kit 项目生成，该文件一般在项目的output目录下。  
 如果使用 MinerU 的命令行调用内置的模型分析，该文件一般在输出路径"/tmp/magic-pdf/pdf-name"下。  
 参考：https://github.com/opendatalab/MinerU/issues/128

-### 10.Linux下报错：Required dependency not installed, please install by "pip install magic-pdf[full-cpu] detectron2 --extra-index-url https://myhloli.github.io/wheels/"
+### 9.报错：Required dependency not installed, please install by "pip install magic-pdf[full-cpu] detectron2 --extra-index-url https://myhloli.github.io/wheels/"

-这种情况可以先使用pip list 检查一下自己的依赖库列表，重点确认下以下几个库有没有安装（版本不一定完全一致，有就可以）
+通过更新0.6.2b1来解决
 ```bash
-opencv-contrib-python     4.6.0.66
-opencv-python             4.6.0.66
-opencv-python-headless    4.10.0.84
-paddleocr                 2.7.3
-paddlepaddle              2.6.1
-torch                     2.2.2
-torchtext                 0.17.2
-torchvision               0.17.2
-```
-如果都有的话，可能是libgl库没有安装，参考 https://github.com/opendatalab/MinerU/issues/165#issuecomment-2245202282 安装libgl库后再试试能不能正常使用。
+pip install magic-pdf[full]==0.6.2b1 -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
\ No newline at end of file
--- a/docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md
+++ b/docs/README_Ubuntu_CUDA_Acceleration_zh_CN.md
+# Ubuntu 22.04 LTS
+
+## 1. 更新apt 
+```bash
+sudo apt-get update
+```
+## 2. 检测是否已安装nvidia驱动
+```bash
+nvidia-smi 
+```
+如果看到类似如下的信息，说明已经安装了nvidia驱动，可以跳过步骤3
+```
+---------------------------------------------------------------------------------------+
+| NVIDIA-SMI 537.34                 Driver Version: 537.34       CUDA Version: 12.2     |
+|-----------------------------------------+----------------------+----------------------+
+| GPU  Name                     TCC/WDDM  | Bus-Id        Disp.A | Volatile Uncorr. ECC |
+| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
+|                                         |                      |               MIG M. |
+|=========================================+======================+======================|
+|   0  NVIDIA GeForce RTX 3060 Ti   WDDM  | 00000000:01:00.0  On |                  N/A |
+|  0%   51C    P8              12W / 200W |   1489MiB /  8192MiB |      5%      Default |
+|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
+```
+## 3. 安装驱动
+如没有驱动，则通过如下命令
+```bash
+sudo apt-get install nvidia-driver-545
+```
+安装专有驱动，安装完成后，重启电脑
+```bash
+reboot
+```
+## 4. 安装anacoda
+如果已安装conda，可以跳过本步骤
+```bash
+wget https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Linux-x86_64.sh
+bash Anaconda3-2024.06-1-Linux-x86_64.sh
+```
+最后一步输入yes，关闭终端重新打开
+## 5. 使用conda 创建环境
+需指定python版本为3.10
+```bash
+conda create -n MinerU python=3.10
+conda activate MinerU
+```
+## 6. 安装应用
+```bash
+pip install magic-pdf[full]==0.6.2b1 detectron2 --extra-index-url https://wheels.myhloli.com -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
+> ❗️下载完成后，务必通过以下命令确认magic-pdf的版本是否正确
+> 
+> ```bash
+> magic-pdf --version
+>```
+> 如果版本号小于0.6.2，请到issue中向我们反馈
+
+## 7. 下载模型
+详细参考 [如何下载模型文件](how_to_download_models_zh_cn.md)  
+下载后请将models目录移动到空间较大的ssd磁盘目录  
+> ❗️模型下载后请务必检查模型文件是否下载完整
+> 
+> 请检查目录下的模型文件大小与网页上描述是否一致，如果可以的话，最好通过sha256校验模型是否下载完整
+> 
+## 8. 第一次运行前的配置
+在仓库根目录可以获得 [magic-pdf.template.json](../magic-pdf.template.json) 配置模版文件
+> ❗️务必执行以下命令将配置文件拷贝到【用户目录】下，否则程序将无法运行
+>  
+> linux用户目录为 "/home/用户名"
+```bash
+wget https://github.com/opendatalab/MinerU/raw/master/magic-pdf.template.json
+cp magic-pdf.template.json ~/magic-pdf.json
+```
+
+在用户目录中找到magic-pdf.json文件并配置"models-dir"为[7. 下载模型](#7-下载模型)中下载的模型权重文件所在目录
+> ❗️务必正确配置模型权重文件所在目录的【绝对路径】，否则会因为找不到模型文件而导致程序无法运行
+> 
+```json
+{
+  "models-dir": "/tmp/models"
+}
+```
+
+## 9. 第一次运行
+从仓库中下载样本文件，并测试
+```bash
+wget https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf
+magic-pdf pdf-command --pdf small_ocr.pdf
+```
+## 10. 测试CUDA加速
+如果您的显卡显存大于等于8G，可以进行以下流程，测试CUDA解析加速效果
+
+**1.修改【用户目录】中配置文件magic-pdf.json中"device-mode"的值**
+```json
+{
+  "device-mode":"cuda"
+}
+```
+**2.运行以下命令测试cuda加速效果**
+```bash
+magic-pdf pdf-command --pdf small_ocr.pdf
+```
+
+## 11. 为ocr开启cuda加速
+> ❗️以下操作需显卡显存大于等于16G才可进行，否则会因为显存不足导致程序崩溃或运行速度下降
+
+**1.下载paddlepaddle-gpu, 安装完成后会自动开启ocr加速**
+```bash
+python -m pip install paddlepaddle-gpu==3.0.0b1 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
+```
+**2.运行以下命令测试ocr加速效果**
+```bash
+magic-pdf pdf-command --pdf small_ocr.pdf
+```
--- a/docs/README_Windows_CUDA_Acceleration_zh_CN.md
+++ b/docs/README_Windows_CUDA_Acceleration_zh_CN.md
+# Windows10/11
+
+## 1. 安装cuda和cuDNN
+
+需要安装的版本 CUDA 11.8 + cuDNN 8.7.0
+- CUDA 11.8 https://developer.nvidia.com/cuda-11-8-0-download-archive
+- cuDNN v8.7.0 (November 28th, 2022), for CUDA 11.x https://developer.nvidia.com/rdp/cudnn-archive
+
+## 2. 安装anaconda
+如果已安装conda，可以跳过本步骤
+
+下载链接：
+https://repo.anaconda.com/archive/Anaconda3-2024.06-1-Windows-x86_64.exe
+
+## 3. 使用conda 创建环境
+需指定python版本为3.10
+```bash
+conda create -n MinerU python=3.10
+conda activate MinerU
+```
+## 4. 安装应用
+```bash
+pip install magic-pdf[full]==0.6.2b1 detectron2 --extra-index-url https://wheels.myhloli.com -i https://pypi.tuna.tsinghua.edu.cn/simple
+```
+> ❗️下载完成后，务必通过以下命令确认magic-pdf的版本是否正确
+> 
+> ```bash
+> magic-pdf --version
+>```
+> 如果版本号小于0.6.2，请到issue中向我们反馈
+
+## 5. 下载模型
+详细参考 [如何下载模型文件](how_to_download_models_zh_cn.md)  
+下载后请将models目录移动到空间较大的ssd磁盘目录  
+> ❗️模型下载后请务必检查模型文件是否下载完整
+> 
+> 请检查目录下的模型文件大小与网页上描述是否一致，如果可以的话，最好通过sha256校验模型是否下载完整
+
+## 6. 第一次运行前的配置
+在仓库根目录可以获得 [magic-pdf.template.json](../magic-pdf.template.json) 配置模版文件
+> ❗️务必执行以下命令将配置文件拷贝到【用户目录】下，否则程序将无法运行
+>  
+> windows用户目录为 "C:\Users\用户名"
+```powershell
+(New-Object System.Net.WebClient).DownloadFile('https://github.com/opendatalab/MinerU/raw/master/magic-pdf.template.json', 'magic-pdf.template.json')
+cp magic-pdf.template.json ~/magic-pdf.json
+```
+
+在用户目录中找到magic-pdf.json文件并配置"models-dir"为[5. 下载模型](#5-下载模型)中下载的模型权重文件所在目录
+> ❗️务必正确配置模型权重文件所在目录的【绝对路径】，否则会因为找不到模型文件而导致程序无法运行
+> 
+> windows系统中此路径应包含盘符，且需把路径中所有的"\"替换为"/",否则会因为转义原因导致json文件语法错误。
+> 
+> 例如：模型放在D盘根目录的models目录，则model-dir的值应为"D:/models"
+```json
+{
+  "models-dir": "/tmp/models"
+}
+```
+
+## 7. 第一次运行
+从仓库中下载样本文件，并测试
+```powershell
+(New-Object System.Net.WebClient).DownloadFile('https://github.com/opendatalab/MinerU/raw/master/demo/small_ocr.pdf', 'small_ocr.pdf')
+magic-pdf pdf-command --pdf small_ocr.pdf
+```
+
+## 8. 测试CUDA加速
+如果您的显卡显存大于等于8G，可以进行以下流程，测试CUDA解析加速效果
+
+**1.覆盖安装支持cuda的torch和torchvision**
+```bash
+pip install --force-reinstall torch==2.3.1 torchvision==0.18.1 --index-url https://download.pytorch.org/whl/cu118
+```
+> ❗️务必在命令中指定以下版本
+> ```bash
+> torch==2.3.1 torchvision==0.18.1 
+> ```
+> 这是我们支持的最高版本，如果不指定版本会自动安装更高版本导致程序无法运行
+
+**2.修改【用户目录】中配置文件magic-pdf.json中"device-mode"的值**
+```json
+{
+  "device-mode":"cuda"
+}
+```
+**3.运行以下命令测试cuda加速效果**
+```bash
+magic-pdf pdf-command --pdf small_ocr.pdf
+```
+
+## 9. 为ocr开启cuda加速
+> ❗️以下操作需显卡显存大于等于16G才可进行，否则会因为显存不足导致程序崩溃或运行速度下降
+
+**1.下载paddlepaddle-gpu, 安装完成后会自动开启ocr加速**
+```bash
+pip install paddlepaddle-gpu==2.6.1
+```
+**2.运行以下命令测试ocr加速效果**
+```bash
+magic-pdf pdf-command --pdf small_ocr.pdf
+```
+
--- a/docs/how_to_download_models_en.md
+++ b/docs/how_to_download_models_en.md
@@ -35,7 +35,7 @@ model_dir = snapshot_download('wanderkid/PDF-Extract-Kit')
 Alternatively, you can use Git to clone the model repository from ModelScope:

 ```bash
-git clone https://www.modelscope.cn/wanderkid/PDF-Extract-Kit.git
+git lfs clone https://www.modelscope.cn/wanderkid/PDF-Extract-Kit.git
 ```


@@ -45,7 +45,7 @@ Put [model files]() here:
 ./
 ├── Layout
 │   ├── config.json
-│   └── weights.pth
+│   └── model_final.pth
 ├── MFD
 │   └── weights.pt
 ├── MFR
@@ -57,4 +57,4 @@ Put [model files]() here:
 │       ├── tokenizer_config.json
 │       └── tokenizer.json
 └── README.md
-```
\ No newline at end of file
+```
--- a/docs/how_to_download_models_zh_cn.md
+++ b/docs/how_to_download_models_zh_cn.md
-### 安装 Git LFS
-开始之前，请确保您的系统上已安装 Git 大文件存储 (Git LFS)。使用以下命令进行安装
+# 如何下载模型文件

-```bash
-git lfs install
-```
+模型文件可以从Hugging Face 或 Model Scope 下载，由于网络原因，国内用户访问HF 可能会失败，请使用 ModelScope。
+
+
+方法一：[从 Hugging Face 下载模型](#方法一从-hugging-face-下载模型)
+
+方法二：[从 ModelScope 下载模型](#方法二从-modelscope-下载模型)

-### 从 Hugging Face 下载模型
-请使用以下命令从 Hugging Face 下载 PDF-Extract-Kit 模型：
+## 方法一：从 Hugging Face 下载模型
+
+使用Git LFS 从Hugging Face下载模型文件

 ```bash
-git lfs clone https://huggingface.co/wanderkid/PDF-Extract-Kit
+git lfs install # 安装 Git 大文件存储插件 (Git LFS) 
+git lfs clone https://huggingface.co/wanderkid/PDF-Extract-Kit # 从 Hugging Face 下载 PDF-Extract-Kit 模型
 ```

-确保在克隆过程中启用了 Git LFS，以便正确下载所有大文件。

+## 方法二：从 ModelScope 下载模型
+ModelScope 支持SDK或模型下载，任选一个即可。
+
+[Git lsf下载](#1利用git-lsf下载)
+
+[SDK下载](#2利用sdk下载)

-### 从 ModelScope 下载模型
+### 1）利用Git lsf下载
+
+```bash
+git lfs install
+git lfs clone https://www.modelscope.cn/wanderkid/PDF-Extract-Kit.git
+```

-#### SDK下载
+### 2）利用SDK下载

 ```bash
 # 首先安装modelscope
@@ -30,42 +44,18 @@ from modelscope import snapshot_download
 model_dir = snapshot_download('wanderkid/PDF-Extract-Kit')
 ```

-#### Git下载
-也可以使用git clone从 ModelScope 下载模型:
-
-需要先安装git lfs
-
->##### On Linux
->
->Debian and RPM packages are available from packagecloud, see the [Linux installation instructions](INSTALLING.md).
->
->##### On macOS
->
->[Homebrew](https://brew.sh) bottles are distributed and can be installed via `brew install git-lfs`.
->
->##### On Windows
->
->Git LFS is included in the distribution of [Git for Windows](https://gitforwindows.org/).
->Alternatively, you can install a recent version of Git LFS from the [Chocolatey](https://chocolatey.org/) package manager.
-
-然后通过git clone下载模型:
-```bash
-git lfs clone https://www.modelscope.cn/wanderkid/PDF-Extract-Kit.git
-```
-
-
-将 'models' 目录移动到具有较大磁盘空间的目录中，最好是在固态硬盘(SSD)上。
-
+## 额外步骤

+### 1.检查模型目录是否下载完整
 模型文件夹的结构如下，包含了不同组件的配置文件和权重文件：
 ```
 ./
-├── Layout
+├── Layout  # 布局检测模型
 │   ├── config.json
 │   └── model_final.pth
-├── MFD
+├── MFD  # 公式检测
 │   └── weights.pt
-├── MFR
+├── MFR  # 公式识别模型
 │   └── UniMERNet
 │       ├── config.json
 │       ├── preprocessor_config.json
@@ -73,7 +63,7 @@ git lfs clone https://www.modelscope.cn/wanderkid/PDF-Extract-Kit.git
 │       ├── README.md
 │       ├── tokenizer_config.json
 │       └── tokenizer.json
-│── TabRec
+│── TabRec # 表格识别模型
 │   └─StructEqTable
 │       ├── config.json
 │       ├── generation_config.json
@@ -85,3 +75,10 @@ git lfs clone https://www.modelscope.cn/wanderkid/PDF-Extract-Kit.git
 │       └── tokenizer_config.json 
 └── README.md
 ```
+
+### 2.检查模型文件是否下载完整
+请检查目录下的模型文件大小与网页上描述是否一致，如果可以的话，最好通过sha256校验模型是否下载完整
+
+### 3.移动模型到固态硬盘
+将 'models' 目录移动到具有较大磁盘空间的目录中，最好是在固态硬盘(SSD)上。
+此外在 `~/magic-pdf.json`里修改模型的目录指向最终的模型存放位置，否则会报模型无法加载的错误。
--- a/docs/images/layout_example.png
+++ b/docs/images/layout_example.png
--- a/docs/images/poly.png
+++ b/docs/images/poly.png
--- a/docs/images/spans_example.png
+++ b/docs/images/spans_example.png
--- a/docs/output_file_zh_cn.md
+++ b/docs/output_file_zh_cn.md
--- a/magic-pdf.template.json
+++ b/magic-pdf.template.json
@@ -3,7 +3,6 @@
        "bucket-name-1":["ak", "sk", "endpoint"],
        "bucket-name-2":["ak", "sk", "endpoint"]
    },
-    "temp-output-dir":"/tmp",
    "models-dir":"/tmp/models",
    "device-mode":"cpu",
    "table-config": {

--- a/magic_pdf/cli/magicpdf.py
+++ b/magic_pdf/cli/magicpdf.py
--- a/magic_pdf/dict2md/ocr_mkcontent.py
+++ b/magic_pdf/dict2md/ocr_mkcontent.py
@@ -168,7 +168,7 @@ def merge_para_with_text(para_block):
                else:
                    content = ocr_escape_special_markdown_char(content)
            elif span_type == ContentType.InlineEquation:
-                content = f"${span['content']}$"
+                content = f" ${span['content']}$ "
            elif span_type == ContentType.InterlineEquation:
                content = f"\n$$\n{span['content']}\n$$\n"


--- a/magic_pdf/libs/config_reader.py
+++ b/magic_pdf/libs/config_reader.py
@@ -57,16 +57,6 @@ def get_bucket_name(path):
    return bucket


-def get_local_dir():
-    config = read_config()
-    local_dir = config.get("temp-output-dir")
-    if local_dir is None:
-        logger.warning(f"'temp-output-dir' not found in {CONFIG_FILE_NAME}, use '/tmp' as default")
-        return "/tmp"
-    else:
-        return local_dir
-
-
 def get_local_models_dir():
    config = read_config()
    models_dir = config.get("models-dir")
@@ -92,5 +82,15 @@ def get_table_recog_config():
    return table_config


+def get_table_recog_config():
+    config = read_config()
+    table_config = config.get("table-config")
+    if table_config is None:
+        logger.warning(f"'table-config' not found in {CONFIG_FILE_NAME}, use 'False' as default")
+        return json.loads('{"is_table_recog_enable": false, "max_time": 400}')
+    else:
+        return table_config
+
+
 if __name__ == "__main__":
    ak, sk, endpoint = get_s3_config("llm-raw")
--- a/magic_pdf/model/pdf_extract_kit.py
+++ b/magic_pdf/model/pdf_extract_kit.py
@@ -3,6 +3,7 @@ import os
 import time
 from pypandoc import convert_text

+
 os.environ['NO_ALBUMENTATIONS_UPDATE'] = '1'  # 禁止albumentations检查更新
 try:
    import cv2
@@ -107,8 +108,8 @@ class CustomPEKModel:
        self.apply_table = self.table_config.get("is_table_recog_enable", False)
        self.apply_ocr = ocr
        logger.info(
-            "DocAnalysis init, this may take some times. apply_layout: {}, apply_formula: {}, apply_ocr: {}".format(
-                self.apply_layout, self.apply_formula, self.apply_ocr
+            "DocAnalysis init, this may take some times. apply_layout: {}, apply_formula: {}, apply_ocr: {}, apply_table: {}".format(
+                self.apply_layout, self.apply_formula, self.apply_ocr, self.apply_table
            )
        )
        assert self.apply_layout, "DocAnalysis must contain layout model."

--- a/magic_pdf/pdf_parse_for_train.py
+++ b/magic_pdf/pdf_parse_for_train.py
--- a/magic_pdf/rw/AbsReaderWriter.py
+++ b/magic_pdf/rw/AbsReaderWriter.py
@@ -2,33 +2,16 @@ from abc import ABC, abstractmethod


 class AbsReaderWriter(ABC):
-    """
-    同时支持二进制和文本读写的抽象类
-    """
    MODE_TXT = "text"
    MODE_BIN = "binary"
-
-    def __init__(self, parent_path):
-        # 初始化代码可以在这里添加，如果需要的话
-        self.parent_path = parent_path # 对于本地目录是父目录，对于s3是会写到这个path下。
-
    @abstractmethod
    def read(self, path: str, mode=MODE_TXT):
-        """
-        无论对于本地还是s3的路径，检查如果path是绝对路径，那么就不再 拼接parent_path, 如果是相对路径就拼接parent_path
-        """
        raise NotImplementedError

    @abstractmethod
    def write(self, content: str, path: str, mode=MODE_TXT):
-        """
-        无论对于本地还是s3的路径，检查如果path是绝对路径，那么就不再 拼接parent_path, 如果是相对路径就拼接parent_path
-        """
        raise NotImplementedError

    @abstractmethod
-    def read_jsonl(self, path: str, byte_start=0, byte_end=None, encoding='utf-8'):
-        """
-        无论对于本地还是s3的路径，检查如果path是绝对路径，那么就不再 拼接parent_path, 如果是相对路径就拼接parent_path
-        """
+    def read_offset(self, path: str, offset=0, limit=None) -> bytes:
        raise NotImplementedError
--- a/magic_pdf/rw/DiskReaderWriter.py
+++ b/magic_pdf/rw/DiskReaderWriter.py
@@ -3,34 +3,29 @@ from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
 from loguru import logger


-MODE_TXT = "text"
-MODE_BIN = "binary"
-
-
 class DiskReaderWriter(AbsReaderWriter):
-
    def __init__(self, parent_path, encoding="utf-8"):
        self.path = parent_path
        self.encoding = encoding

-    def read(self, path, mode=MODE_TXT):
+    def read(self, path, mode=AbsReaderWriter.MODE_TXT):
        if os.path.isabs(path):
            abspath = path
        else:
            abspath = os.path.join(self.path, path)
        if not os.path.exists(abspath):
-            logger.error(f"文件 {abspath} 不存在")
-            raise Exception(f"文件 {abspath} 不存在")
-        if mode == MODE_TXT:
+            logger.error(f"file {abspath} not exists")
+            raise Exception(f"file {abspath} no exists")
+        if mode == AbsReaderWriter.MODE_TXT:
            with open(abspath, "r", encoding=self.encoding) as f:
                return f.read()
-        elif mode == MODE_BIN:
+        elif mode == AbsReaderWriter.MODE_BIN:
            with open(abspath, "rb") as f:
                return f.read()
        else:
            raise ValueError("Invalid mode. Use 'text' or 'binary'.")

-    def write(self, content, path, mode=MODE_TXT):
+    def write(self, content, path, mode=AbsReaderWriter.MODE_TXT):
        if os.path.isabs(path):
            abspath = path
        else:
@@ -38,29 +33,42 @@ class DiskReaderWriter(AbsReaderWriter):
        directory_path = os.path.dirname(abspath)
        if not os.path.exists(directory_path):
            os.makedirs(directory_path)
-        if mode == MODE_TXT:
+        if mode == AbsReaderWriter.MODE_TXT:
            with open(abspath, "w", encoding=self.encoding, errors="replace") as f:
                f.write(content)

-        elif mode == MODE_BIN:
+        elif mode == AbsReaderWriter.MODE_BIN:
            with open(abspath, "wb") as f:
                f.write(content)
        else:
            raise ValueError("Invalid mode. Use 'text' or 'binary'.")

-    def read_jsonl(self, path: str, byte_start=0, byte_end=None, encoding="utf-8"):
-        return self.read(path)
+    def read_offset(self, path: str, offset=0, limit=None):
+        abspath = path
+        if not os.path.isabs(path):
+            abspath = os.path.join(self.path, path)
+        with open(abspath, "rb") as f:
+            f.seek(offset)
+            return f.read(limit)


-# 使用示例
 if __name__ == "__main__":
-    file_path = "io/test/example.txt"
-    drw = DiskReaderWriter("D:\projects\papayfork\Magic-PDF\magic_pdf")
+    if 0:
+        file_path = "io/test/example.txt"
+        drw = DiskReaderWriter("D:\projects\papayfork\Magic-PDF\magic_pdf")
+
+        # 写入内容到文件
+        drw.write(b"Hello, World!", path="io/test/example.txt", mode="binary")
+
+        # 从文件读取内容
+        content = drw.read(path=file_path)
+        if content:
+            logger.info(f"从 {file_path} 读取的内容: {content}")
+    if 1:
+        drw = DiskReaderWriter("/opt/data/pdf/resources/test/io/")
+        content_bin = drw.read_offset("1.txt")
+        assert content_bin == b"ABCD!"

-    # 写入内容到文件
-    drw.write(b"Hello, World!", path="io/test/example.txt", mode="binary")
+        content_bin = drw.read_offset("1.txt", offset=1, limit=2)
+        assert content_bin == b"BC"

-    # 从文件读取内容
-    content = drw.read(path=file_path)
-    if content:
-        logger.info(f"从 {file_path} 读取的内容: {content}")
--- a/magic_pdf/rw/S3ReaderWriter.py
+++ b/magic_pdf/rw/S3ReaderWriter.py
@@ -2,16 +2,18 @@ from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
 from magic_pdf.libs.commons import parse_aws_param, parse_bucket_key, join_path
 import boto3
 from loguru import logger
-from boto3.s3.transfer import TransferConfig
 from botocore.config import Config
-import os
-
-MODE_TXT = "text"
-MODE_BIN = "binary"


 class S3ReaderWriter(AbsReaderWriter):
-    def __init__(self, ak: str, sk: str, endpoint_url: str, addressing_style: str = 'auto', parent_path: str = ''):
+    def __init__(
+        self,
+        ak: str,
+        sk: str,
+        endpoint_url: str,
+        addressing_style: str = "auto",
+        parent_path: str = "",
+    ):
        self.client = self._get_client(ak, sk, endpoint_url, addressing_style)
        self.path = parent_path

@@ -21,12 +23,14 @@ class S3ReaderWriter(AbsReaderWriter):
            aws_access_key_id=ak,
            aws_secret_access_key=sk,
            endpoint_url=endpoint_url,
-            config=Config(s3={"addressing_style": addressing_style},
-                          retries={'max_attempts': 5, 'mode': 'standard'}),
+            config=Config(
+                s3={"addressing_style": addressing_style},
+                retries={"max_attempts": 5, "mode": "standard"},
+            ),
        )
        return s3_client

-    def read(self, s3_relative_path, mode=MODE_TXT, encoding="utf-8"):
+    def read(self, s3_relative_path, mode=AbsReaderWriter.MODE_TXT, encoding="utf-8"):
        if s3_relative_path.startswith("s3://"):
            s3_path = s3_relative_path
        else:
@@ -34,22 +38,22 @@ class S3ReaderWriter(AbsReaderWriter):
        bucket_name, key = parse_bucket_key(s3_path)
        res = self.client.get_object(Bucket=bucket_name, Key=key)
        body = res["Body"].read()
-        if mode == MODE_TXT:
+        if mode == AbsReaderWriter.MODE_TXT:
            data = body.decode(encoding)  # Decode bytes to text
-        elif mode == MODE_BIN:
+        elif mode == AbsReaderWriter.MODE_BIN:
            data = body
        else:
            raise ValueError("Invalid mode. Use 'text' or 'binary'.")
        return data

-    def write(self, content, s3_relative_path, mode=MODE_TXT, encoding="utf-8"):
+    def write(self, content, s3_relative_path, mode=AbsReaderWriter.MODE_TXT, encoding="utf-8"):
        if s3_relative_path.startswith("s3://"):
            s3_path = s3_relative_path
        else:
            s3_path = join_path(self.path, s3_relative_path)
-        if mode == MODE_TXT:
+        if mode == AbsReaderWriter.MODE_TXT:
            body = content.encode(encoding)  # Encode text data as bytes
-        elif mode == MODE_BIN:
+        elif mode == AbsReaderWriter.MODE_BIN:
            body = content
        else:
            raise ValueError("Invalid mode. Use 'text' or 'binary'.")
@@ -57,51 +61,82 @@ class S3ReaderWriter(AbsReaderWriter):
        self.client.put_object(Body=body, Bucket=bucket_name, Key=key)
        logger.info(f"内容已写入 {s3_path} ")

-    def read_jsonl(self, path: str, byte_start=0, byte_end=None, mode=MODE_TXT, encoding='utf-8'):
+    def read_offset(self, path: str, offset=0, limit=None) -> bytes:
        if path.startswith("s3://"):
            s3_path = path
        else:
            s3_path = join_path(self.path, path)
        bucket_name, key = parse_bucket_key(s3_path)

-        range_header = f'bytes={byte_start}-{byte_end}' if byte_end else f'bytes={byte_start}-'
+        range_header = (
+            f"bytes={offset}-{offset+limit-1}" if limit else f"bytes={offset}-"
+        )
        res = self.client.get_object(Bucket=bucket_name, Key=key, Range=range_header)
-        body = res["Body"].read()
-        if mode == MODE_TXT:
-            data = body.decode(encoding)  # Decode bytes to text
-        elif mode == MODE_BIN:
-            data = body
-        else:
-            raise ValueError("Invalid mode. Use 'text' or 'binary'.")
-        return data
+        return res["Body"].read()


 if __name__ == "__main__":
-    # Config the connection info
-    ak = ""
-    sk = ""
-    endpoint_url = ""
-    addressing_style = "auto"
-    bucket_name = ""
-    # Create an S3ReaderWriter object
-    s3_reader_writer = S3ReaderWriter(ak, sk, endpoint_url, addressing_style, "s3://bucket_name/")
+    if 0:
+        # Config the connection info
+        ak = ""
+        sk = ""
+        endpoint_url = ""
+        addressing_style = "auto"
+        bucket_name = ""
+        # Create an S3ReaderWriter object
+        s3_reader_writer = S3ReaderWriter(
+            ak, sk, endpoint_url, addressing_style, "s3://bucket_name/"
+        )

-    # Write text data to S3
-    text_data = "This is some text data"
-    s3_reader_writer.write(data=text_data, s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=MODE_TXT)
+        # Write text data to S3
+        text_data = "This is some text data"
+        s3_reader_writer.write(
+            text_data,
+            s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json",
+            mode=AbsReaderWriter.MODE_TXT,
+        )
+
+        # Read text data from S3
+        text_data_read = s3_reader_writer.read(
+            s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=AbsReaderWriter.MODE_TXT
+        )
+        logger.info(f"Read text data from S3: {text_data_read}")
+        # Write binary data to S3
+        binary_data = b"This is some binary data"
+        s3_reader_writer.write(
+            text_data,
+            s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json",
+            mode=AbsReaderWriter.MODE_BIN,
+        )

-    # Read text data from S3
-    text_data_read = s3_reader_writer.read(s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=MODE_TXT)
-    logger.info(f"Read text data from S3: {text_data_read}")
-    # Write binary data to S3
-    binary_data = b"This is some binary data"
-    s3_reader_writer.write(data=text_data, s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=MODE_BIN)
+        # Read binary data from S3
+        binary_data_read = s3_reader_writer.read(
+            s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=AbsReaderWriter.MODE_BIN
+        )
+        logger.info(f"Read binary data from S3: {binary_data_read}")
+
+        # Range Read text data from S3
+        binary_data_read = s3_reader_writer.read_offset(
+            path=f"s3://{bucket_name}/ebook/test/test.json", offset=0, limit=10
+        )
+        logger.info(f"Read binary data from S3: {binary_data_read}")
+    if 1:
+        import os
+        import json

-    # Read binary data from S3
-    binary_data_read = s3_reader_writer.read(s3_relative_path=f"s3://{bucket_name}/ebook/test/test.json", mode=MODE_BIN)
-    logger.info(f"Read binary data from S3: {binary_data_read}")
+        ak = os.getenv("AK", "")
+        sk = os.getenv("SK", "")
+        endpoint_url = os.getenv("ENDPOINT", "")
+        bucket = os.getenv("S3_BUCKET", "")
+        prefix = os.getenv("S3_PREFIX", "")
+        key_basename = os.getenv("S3_KEY_BASENAME", "")
+        s3_reader_writer = S3ReaderWriter(
+            ak, sk, endpoint_url, "auto", f"s3://{bucket}/{prefix}"
+        )
+        content_bin = s3_reader_writer.read_offset(key_basename)
+        assert content_bin[:10] == b'{"track_id'
+        assert content_bin[-10:] == b'r":null}}\n'

-    # Range Read text data from S3
-    binary_data_read = s3_reader_writer.read_jsonl(path=f"s3://{bucket_name}/ebook/test/test.json",
-                                                   byte_start=0, byte_end=10, mode=MODE_BIN)
-    logger.info(f"Read binary data from S3: {binary_data_read}")
+        content_bin = s3_reader_writer.read_offset(key_basename, offset=424, limit=426)
+        jso = json.dumps(content_bin.decode("utf-8"))
+        print(jso)
--- a/magic_pdf/cli/__init__.py
+++ b/magic_pdf/cli/__init__.py
--- a/magic_pdf/tools/cli.py
+++ b/magic_pdf/tools/cli.py
+import os
+import click
+from loguru import logger
+from pathlib import Path
+
+from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
+from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
+import magic_pdf.model as model_config
+from magic_pdf.tools.common import parse_pdf_methods, do_parse
+from magic_pdf.libs.version import __version__
+
+
+@click.command()
+@click.version_option(__version__, "--version", "-v", help="display the version and exit")
+@click.option(
+    "-p",
+    "--path",
+    "path",
+    type=click.Path(exists=True),
+    required=True,
+    help="local pdf filepath or directory",
+)
+@click.option(
+    "-o",
+    "--output-dir",
+    "output_dir",
+    type=str,
+    help="output local directory",
+    default="",
+)
+@click.option(
+    "-m",
+    "--method",
+    "method",
+    type=parse_pdf_methods,
+    help="""the method for parsing pdf. 
+ocr: using ocr technique to extract information from pdf.
+txt: suitable for the text-based pdf only and outperform ocr.
+auto: automatically choose the best method for parsing pdf from ocr and txt.
+without method specified, auto will be used by default.""",
+    default="auto",
+)
+def cli(path, output_dir, method):
+    model_config.__use_inside_model__ = True
+    model_config.__model_mode__ = "full"
+    if output_dir == "":
+        if os.path.isdir(path):
+            output_dir = os.path.join(path, "output")
+        else:
+            output_dir = os.path.join(os.path.dirname(path), "output")
+
+    def read_fn(path):
+        disk_rw = DiskReaderWriter(os.path.dirname(path))
+        return disk_rw.read(os.path.basename(path), AbsReaderWriter.MODE_BIN)
+
+    def parse_doc(doc_path: str):
+        try:
+            file_name = str(Path(doc_path).stem)
+            pdf_data = read_fn(doc_path)
+            do_parse(
+                output_dir,
+                file_name,
+                pdf_data,
+                [],
+                method,
+            )
+
+        except Exception as e:
+            logger.exception(e)
+
+    if os.path.isdir(path):
+        for doc_path in Path(path).glob("*.pdf"):
+            parse_doc(doc_path)
+    else:
+        parse_doc(path)
+
+
+if __name__ == "__main__":
+    cli()
--- a/magic_pdf/tools/cli_dev.py
+++ b/magic_pdf/tools/cli_dev.py
+import os
+import json as json_parse
+import click
+from pathlib import Path
+from magic_pdf.libs.path_utils import (
+    parse_s3path,
+    parse_s3_range_params,
+    remove_non_official_s3_args,
+)
+from magic_pdf.libs.config_reader import (
+    get_s3_config,
+)
+from magic_pdf.rw.S3ReaderWriter import S3ReaderWriter
+from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
+from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
+import magic_pdf.model as model_config
+from magic_pdf.tools.common import parse_pdf_methods, do_parse
+from magic_pdf.libs.version import __version__
+
+
+def read_s3_path(s3path):
+    bucket, key = parse_s3path(s3path)
+
+    s3_ak, s3_sk, s3_endpoint = get_s3_config(bucket)
+    s3_rw = S3ReaderWriter(
+        s3_ak, s3_sk, s3_endpoint, "auto", remove_non_official_s3_args(s3path)
+    )
+    may_range_params = parse_s3_range_params(s3path)
+    if may_range_params is None or 2 != len(may_range_params):
+        byte_start, byte_end = 0, None
+    else:
+        byte_start, byte_end = int(may_range_params[0]), int(may_range_params[1])
+        byte_end += byte_start - 1
+    return s3_rw.read_jsonl(
+        remove_non_official_s3_args(s3path),
+        byte_start,
+        byte_end,
+        AbsReaderWriter.MODE_BIN,
+    )
+
+
+@click.group()
+@click.version_option(__version__, "--version", "-v", help="显示版本信息")
+def cli():
+    pass
+
+
+@cli.command()
+@click.option(
+    "-j",
+    "--jsonl",
+    "jsonl",
+    type=str,
+    help="输入 jsonl 路径，本地或者 s3 上的文件",
+    required=True,
+)
+@click.option(
+    "-m",
+    "--method",
+    "method",
+    type=parse_pdf_methods,
+    help="指定解析方法。txt: 文本型 pdf 解析方法， ocr: 光学识别解析 pdf, auto: 程序智能选择解析方法",
+    default="auto",
+)
+@click.option(
+    "-o",
+    "--output-dir",
+    "output_dir",
+    type=str,
+    help="输出到本地目录",
+    default="",
+)
+def jsonl(jsonl, method, output_dir):
+    print("haha")
+    model_config.__use_inside_model__ = False
+    full_jsonl_path = os.path.realpath(jsonl)
+    if output_dir == "":
+        output_dir = os.path.join(os.path.dirname(full_jsonl_path), "output")
+
+    if jsonl.startswith("s3://"):
+        jso = json_parse.loads(read_s3_path(jsonl).decode("utf-8"))
+    else:
+        with open(jsonl) as f:
+            jso = json_parse.loads(f.readline())
+    s3_file_path = jso.get("file_location")
+    if s3_file_path is None:
+        s3_file_path = jso.get("path")
+    pdf_file_name = Path(s3_file_path).stem
+    pdf_data = read_s3_path(s3_file_path)
+
+
+    print(pdf_file_name, jso, method)
+    do_parse(
+        output_dir,
+        pdf_file_name,
+        pdf_data,
+        jso["doc_layout_result"],
+        method,
+        f_dump_content_list=True,
+    )
+
+
+@cli.command()
+@click.option(
+    "-p",
+    "--pdf",
+    "pdf",
+    type=click.Path(exists=True),
+    required=True,
+    help="本地 PDF 文件",
+)
+@click.option(
+    "-j",
+    "--json",
+    "json_data",
+    type=click.Path(exists=True),
+    required=True,
+    help="本地模型推理出的 json 数据",
+)
+@click.option(
+    "-o", "--output-dir", "output_dir", type=str, help="本地输出目录", default=""
+)
+@click.option(
+    "-m",
+    "--method",
+    "method",
+    type=parse_pdf_methods,
+    help="指定解析方法。txt: 文本型 pdf 解析方法， ocr: 光学识别解析 pdf, auto: 程序智能选择解析方法",
+    default="auto",
+)
+def pdf(pdf, json_data, output_dir, method):
+    model_config.__use_inside_model__ = False
+    full_pdf_path = os.path.realpath(pdf)
+    if output_dir == "":
+        output_dir = os.path.join(os.path.dirname(full_pdf_path), "output")
+
+    def read_fn(path):
+        disk_rw = DiskReaderWriter(os.path.dirname(path))
+        return disk_rw.read(os.path.basename(path), AbsReaderWriter.MODE_BIN)
+
+    model_json_list = json_parse.loads(read_fn(json_data).decode("utf-8"))
+
+    file_name = str(Path(full_pdf_path).stem)
+    pdf_data = read_fn(full_pdf_path)
+    do_parse(
+        output_dir,
+        file_name,
+        pdf_data,
+        model_json_list,
+        method,
+        f_dump_content_list=True,
+    )
+
+
+if __name__ == "__main__":
+    cli()
--- a/magic_pdf/tools/common.py
+++ b/magic_pdf/tools/common.py
+import os
+import json as json_parse
+import copy
+import click
+from loguru import logger
+from magic_pdf.libs.MakeContentConfig import DropMode, MakeMode
+from magic_pdf.libs.draw_bbox import draw_layout_bbox, draw_span_bbox
+from magic_pdf.pipe.UNIPipe import UNIPipe
+from magic_pdf.pipe.OCRPipe import OCRPipe
+from magic_pdf.pipe.TXTPipe import TXTPipe
+from magic_pdf.rw.DiskReaderWriter import DiskReaderWriter
+from magic_pdf.rw.AbsReaderWriter import AbsReaderWriter
+import magic_pdf.model as model_config
+
+
+def prepare_env(output_dir, pdf_file_name, method):
+    local_parent_dir = os.path.join(output_dir, pdf_file_name, method)
+
+    local_image_dir = os.path.join(str(local_parent_dir), "images")
+    local_md_dir = local_parent_dir
+    os.makedirs(local_image_dir, exist_ok=True)
+    os.makedirs(local_md_dir, exist_ok=True)
+    return local_image_dir, local_md_dir
+
+
+def do_parse(
+    output_dir,
+    pdf_file_name,
+    pdf_bytes,
+    model_list,
+    parse_method,
+    f_draw_span_bbox=True,
+    f_draw_layout_bbox=True,
+    f_dump_md=True,
+    f_dump_middle_json=True,
+    f_dump_model_json=True,
+    f_dump_orig_pdf=True,
+    f_dump_content_list=False,
+    f_make_md_mode=MakeMode.MM_MD,
+):
+    orig_model_list = copy.deepcopy(model_list)
+    local_image_dir, local_md_dir = prepare_env(output_dir, pdf_file_name, parse_method)
+
+    image_writer, md_writer = DiskReaderWriter(local_image_dir), DiskReaderWriter(
+        local_md_dir
+    )
+    image_dir = str(os.path.basename(local_image_dir))
+
+    if parse_method == "auto":
+        jso_useful_key = {"_pdf_type": "", "model_list": model_list}
+        pipe = UNIPipe(pdf_bytes, jso_useful_key, image_writer, is_debug=True)
+    elif parse_method == "txt":
+        pipe = TXTPipe(pdf_bytes, model_list, image_writer, is_debug=True)
+    elif parse_method == "ocr":
+        pipe = OCRPipe(pdf_bytes, model_list, image_writer, is_debug=True)
+    else:
+        logger.error("unknown parse method")
+        exit(1)
+
+    pipe.pipe_classify()
+
+    if len(model_list) == 0:
+        if model_config.__use_inside_model__:
+            pipe.pipe_analyze()
+            orig_model_list = copy.deepcopy(pipe.model_list)
+        else:
+            logger.error("need model list input")
+            exit(2)
+
+    pipe.pipe_parse()
+    pdf_info = pipe.pdf_mid_data["pdf_info"]
+    if f_draw_layout_bbox:
+        draw_layout_bbox(pdf_info, pdf_bytes, local_md_dir)
+    if f_draw_span_bbox:
+        draw_span_bbox(pdf_info, pdf_bytes, local_md_dir)
+
+    md_content = pipe.pipe_mk_markdown(
+        image_dir, drop_mode=DropMode.NONE, md_make_mode=f_make_md_mode
+    )
+    if f_dump_md:
+        md_writer.write(
+            content=md_content,
+            path=f"{pdf_file_name}.md",
+            mode=AbsReaderWriter.MODE_TXT,
+        )
+
+    if f_dump_middle_json:
+        md_writer.write(
+            content=json_parse.dumps(pipe.pdf_mid_data, ensure_ascii=False, indent=4),
+            path="middle.json",
+            mode=AbsReaderWriter.MODE_TXT,
+        )
+
+    if f_dump_model_json:
+        md_writer.write(
+            content=json_parse.dumps(orig_model_list, ensure_ascii=False, indent=4),
+            path="model.json",
+            mode=AbsReaderWriter.MODE_TXT,
+        )
+
+    if f_dump_orig_pdf:
+        md_writer.write(
+            content=pdf_bytes,
+            path="origin.pdf",
+            mode=AbsReaderWriter.MODE_BIN,
+        )
+
+    content_list = pipe.pipe_mk_uni_format(image_dir, drop_mode=DropMode.NONE)
+    if f_dump_content_list:
+        md_writer.write(
+            content=json_parse.dumps(content_list, ensure_ascii=False, indent=4),
+            path="content_list.json",
+            mode=AbsReaderWriter.MODE_TXT,
+        )
+
+    logger.info(f"local output dir is {local_md_dir}")
+
+
+parse_pdf_methods = click.Choice(["ocr", "txt", "auto"])
--- a/magic_pdf/train_utils/convert_to_train_format.py
+++ b/magic_pdf/train_utils/convert_to_train_format.py
-def convert_to_train_format(jso: dict) -> []:
-    pages = []
-    for k, v in jso.items():
-        if not k.startswith("page_"):
-            continue
-        page_idx = v["page_idx"]
-        width, height = v["page_size"]
-
-        info = {"page_info": {"page_no": page_idx, "height": height, "width": width}}
-
-        bboxes: list[dict] = []
-        for img_bbox in v["image_bboxes_with_caption"]:
-            bbox = {"category_id": 1, "bbox": img_bbox["bbox"]}
-            if "caption" in img_bbox:
-                bbox["caption_bbox"] = img_bbox["caption"]
-            bboxes.append(bbox)
-
-        for tbl_bbox in v["table_bboxes_with_caption"]:
-            bbox = {"category_id": 7, "bbox": tbl_bbox["bbox"]}
-            if "caption" in tbl_bbox:
-                bbox["caption_bbox"] = tbl_bbox["caption"]
-            bboxes.append(bbox)
-
-        for bbox in v["bak_page_no_bboxes"]:
-            n_bbox = {"category_id": 4, "bbox": bbox}
-            bboxes.append(n_bbox)
-
-        for bbox in v["bak_header_bboxes"]:
-            n_bbox = {"category_id": 3, "bbox": bbox}
-            bboxes.append(n_bbox)
-
-        for bbox in v["bak_footer_bboxes"]:
-            n_bbox = {"category_id": 6, "bbox": bbox}
-            bboxes.append(n_bbox)
-
-        # 脚注， 目前没有看到例子
-        for para in v["para_blocks"]:
-            if "paras" in para:
-                paras = para["paras"]
-                for para_key, para_content in paras.items():
-                    para_bbox = para_content["para_bbox"]
-                    is_para_title = para_content["is_para_title"]
-                    if is_para_title:
-                        n_bbox = {"category_id": 0, "bbox": para_bbox}
-                    else:
-                        n_bbox = {"category_id": 2, "bbox": para_bbox}
-                    bboxes.append(n_bbox)
-
-        for inline_equation in v["inline_equations"]:
-            n_bbox = {"category_id": 13, "bbox": inline_equation["bbox"]}
-            bboxes.append(n_bbox)
-
-        for inter_equation in v["interline_equations"]:
-            n_bbox = {"category_id": 10, "bbox": inter_equation["bbox"]}
-            bboxes.append(n_bbox)
-
-        for footnote_bbox in v["bak_footer_note_bboxes"]:
-            n_bbox = {"category_id": 5, "bbox": list(footnote_bbox)}
-            bboxes.append(n_bbox)
-
-        info["bboxes"] = bboxes
-        info["layout_tree"] = v["layout_bboxes"]
-        pages.append(info)
-
-    return pages
--- a/magic_pdf/train_utils/extract_caption.py
+++ b/magic_pdf/train_utils/extract_caption.py
-from magic_pdf.libs.boxbase import _is_in
-
-
-def extract_caption_bbox(outer: list, inner: list) -> list:
-    """
-    ret: list of {
-                    "bbox": [1,2,3,4],
-                    "caption": [5,6,7,8] # may existed
-                }
-
-    """
-    found_count = 0  # for debug
-    print(outer, inner)
-
-    def is_float_equal(a, b):
-        if 0.01 > abs(a - b):  # non strict float equal compare
-            return True
-        return False
-
-    outer_h = {i: outer[i] for i in range(len(outer))}
-    ret = []
-    for v in inner:
-        ix0, iy0, ix1, iy1 = v
-        found_idx = None
-        d = {"bbox": v[:4]}
-        for k in outer_h:
-            ox0, oy0, ox1, oy1 = outer_h[k]
-            equal_float_flags = [
-                is_float_equal(ix0, ox0),
-                is_float_equal(iy0, oy0),
-                is_float_equal(ix1, ox1),
-                is_float_equal(iy1, oy1),
-            ]
-            if _is_in(v, outer_h[k]) and not all(equal_float_flags):
-                found_idx = k
-                break
-        if found_idx is not None:
-            found_count += 1
-            captions: list[list] = []
-            ox0, oy0, ox1, oy1 = outer_h[found_idx]
-            captions = [
-                [ox0, oy0, ix0, oy1],
-                [ox0, oy0, ox1, iy0],
-                [ox0, iy1, ox1, oy1],
-                [ix1, oy0, ox1, oy1],
-            ]
-            captions = sorted(
-                captions,
-                key=lambda rect: abs(rect[0] - rect[2]) * abs(rect[1] - rect[3]),
-            )  # 面积最大的框就是caption
-            d["caption"] = captions[-1]
-            outer_h.pop(
-                found_idx
-            )  # 同一个 outer box 只能用于确定一个 inner box 的 caption 位置。
-
-        ret.append(d)
-
-    print("found_count: ", found_count)
-    return ret
--- a/magic_pdf/train_utils/remove_footer_header.py
+++ b/magic_pdf/train_utils/remove_footer_header.py
-import re
-
-from magic_pdf.libs.boxbase import _is_in_or_part_overlap
-from magic_pdf.libs.drop_tag import CONTENT_IN_FOOT_OR_HEADER, PAGE_NO
-
-
-"""
-    copy from pre_proc/remove_footer_header.py
-"""
-
-
-def remove_headder_footer_one_page(
-    text_raw_blocks,
-    image_bboxes,
-    table_bboxes,
-    header_bboxs,
-    footer_bboxs,
-    page_no_bboxs,
-    page_w,
-    page_h,
-):
-    """
-    删除页眉页脚，页码
-    从line级别进行删除，删除之后观察这个text-block是否是空的，如果是空的，则移动到remove_list中
-    """
-    if 1:
-        return image_bboxes, table_bboxes, text_raw_blocks, [], [], []
-
-    header = []
-    footer = []
-    if len(header) == 0:
-        model_header = header_bboxs
-        if model_header:
-            x0 = min([x for x, _, _, _ in model_header])
-            y0 = min([y for _, y, _, _ in model_header])
-            x1 = max([x1 for _, _, x1, _ in model_header])
-            y1 = max([y1 for _, _, _, y1 in model_header])
-            header = [x0, y0, x1, y1]
-    if len(footer) == 0:
-        model_footer = footer_bboxs
-        if model_footer:
-            x0 = min([x for x, _, _, _ in model_footer])
-            y0 = min([y for _, y, _, _ in model_footer])
-            x1 = max([x1 for _, _, x1, _ in model_footer])
-            y1 = max([y1 for _, _, _, y1 in model_footer])
-            footer = [x0, y0, x1, y1]
-
-    header_y0 = 0 if len(header) == 0 else header[3]
-    footer_y0 = page_h if len(footer) == 0 else footer[1]
-    if page_no_bboxs:
-        top_part = [b for b in page_no_bboxs if b[3] < page_h / 2]
-        btn_part = [b for b in page_no_bboxs if b[1] > page_h / 2]
-
-        top_max_y0 = max([b[1] for b in top_part]) if top_part else 0
-        btn_min_y1 = min([b[3] for b in btn_part]) if btn_part else page_h
-
-        header_y0 = max(header_y0, top_max_y0)
-        footer_y0 = min(footer_y0, btn_min_y1)
-
-    content_boundry = [0, header_y0, page_w, footer_y0]
-
-    header = [0, 0, page_w, header_y0]
-    footer = [0, footer_y0, page_w, page_h]
-
-    """以上计算出来了页眉页脚的边界，下面开始进行删除"""
-    text_block_to_remove = []
-    # 首先检查每个textblock
-    for blk in text_raw_blocks:
-        if len(blk["lines"]) > 0:
-            for line in blk["lines"]:
-                line_del = []
-                for span in line["spans"]:
-                    span_del = []
-                    if span["bbox"][3] < header_y0:
-                        span_del.append(span)
-                    elif _is_in_or_part_overlap(
-                        span["bbox"], header
-                    ) or _is_in_or_part_overlap(span["bbox"], footer):
-                        span_del.append(span)
-                for span in span_del:
-                    line["spans"].remove(span)
-                if not line["spans"]:
-                    line_del.append(line)
-
-            for line in line_del:
-                blk["lines"].remove(line)
-        else:
-            # if not blk['lines']:
-            blk["tag"] = CONTENT_IN_FOOT_OR_HEADER
-            text_block_to_remove.append(blk)
-
-    """有的时候由于pageNo太小了，总是会有一点和content_boundry重叠一点，被放入正文，因此对于pageNo，进行span粒度的删除"""
-    page_no_block_2_remove = []
-    if page_no_bboxs:
-        for pagenobox in page_no_bboxs:
-            for block in text_raw_blocks:
-                if _is_in_or_part_overlap(
-                    pagenobox, block["bbox"]
-                ):  # 在span级别删除页码
-                    for line in block["lines"]:
-                        for span in line["spans"]:
-                            if _is_in_or_part_overlap(pagenobox, span["bbox"]):
-                                # span['text'] = ''
-                                span["tag"] = PAGE_NO
-                                # 检查这个block是否只有这一个span，如果是，那么就把这个block也删除
-                                if len(line["spans"]) == 1 and len(block["lines"]) == 1:
-                                    page_no_block_2_remove.append(block)
-    else:
-        # 测试最后一个是不是页码：规则是，最后一个block仅有1个line,一个span,且text是数字，空格，符号组成，不含字母,并且包含数字
-        if len(text_raw_blocks) > 0:
-            text_raw_blocks.sort(key=lambda x: x["bbox"][1], reverse=True)
-            last_block = text_raw_blocks[0]
-            if len(last_block["lines"]) == 1:
-                last_line = last_block["lines"][0]
-                if len(last_line["spans"]) == 1:
-                    last_span = last_line["spans"][0]
-                    if (
-                        last_span["text"].strip()
-                        and not re.search("[a-zA-Z]", last_span["text"])
-                        and re.search("[0-9]", last_span["text"])
-                    ):
-                        last_span["tag"] = PAGE_NO
-                        page_no_block_2_remove.append(last_block)
-
-    for b in page_no_block_2_remove:
-        text_block_to_remove.append(b)
-
-    for blk in text_block_to_remove:
-        if blk in text_raw_blocks:
-            text_raw_blocks.remove(blk)
-
-    text_block_remain = text_raw_blocks
-    image_bbox_to_remove = [
-        bbox
-        for bbox in image_bboxes
-        if not _is_in_or_part_overlap(bbox, content_boundry)
-    ]
-
-    image_bbox_remain = [
-        bbox for bbox in image_bboxes if _is_in_or_part_overlap(bbox, content_boundry)
-    ]
-    table_bbox_to_remove = [
-        bbox
-        for bbox in table_bboxes
-        if not _is_in_or_part_overlap(bbox, content_boundry)
-    ]
-    table_bbox_remain = [
-        bbox for bbox in table_bboxes if _is_in_or_part_overlap(bbox, content_boundry)
-    ]
-
-    #        1,                 2,                3
-    return (
-        image_bbox_remain,
-        table_bbox_remain,
-        text_block_remain,
-        text_block_to_remove,
-        image_bbox_to_remove,
-        table_bbox_to_remove,
-    )
--- a/magic_pdf/train_utils/vis_utils.py
+++ b/magic_pdf/train_utils/vis_utils.py
--- a/setup.py
+++ b/setup.py
@@ -36,12 +36,15 @@ if __name__ == '__main__':
                     "paddlepaddle==3.0.0b1;platform_system=='Linux'",
                     "paddlepaddle==2.6.1;platform_system=='Windows' or platform_system=='Darwin'",
                     ],
-            "full": ["unimernet==0.1.6",
-                     "matplotlib",
-                     "ultralytics",
-                     "paddleocr==2.7.3",
-                     "paddlepaddle==3.0.0b1;platform_system=='Linux'",
-                     "paddlepaddle==2.6.1;platform_system=='Windows' or platform_system=='Darwin'",
+            "full": ["unimernet==0.1.6",  # 0.1.6版本大幅裁剪依赖包范围，推荐使用此版本
+                     "matplotlib<=3.9.0;platform_system=='Windows'",  # 3.9.1及之后不提供windows的预编译包，避免一些没有编译环境的windows设备安装失败
+                     "matplotlib;platform_system=='Linux' or platform_system=='Darwin'",  # linux 和 macos 不应限制matplotlib的最高版本，以避免无法更新导致的一些bug
+                     "ultralytics",  # yolov8,公式检测
+                     "paddleocr==2.7.3",  # 2.8.0及2.8.1版本与detectron2有冲突，需锁定2.7.3
+                     "paddlepaddle==3.0.0b1;platform_system=='Linux'",  # 解决linux的段异常问题
+                     "paddlepaddle==2.6.1;platform_system=='Windows' or platform_system=='Darwin'",  # windows版本3.0.0b1效率下降，需锁定2.6.1
+                     "pypandoc",  # 表格解析latex转html
+                     "struct-eqtable==0.1.0",  # 表格解析
                     "detectron2"
                     ],
        },
@@ -52,7 +55,8 @@ if __name__ == '__main__':
        python_requires=">=3.9",  # 项目依赖的 Python 版本
        entry_points={
            "console_scripts": [
-                "magic-pdf = magic_pdf.cli.magicpdf:cli"
+                "magic-pdf = magic_pdf.tools.cli:cli",
+                "magic-pdf-dev = magic_pdf.tools.cli_dev:cli" 
            ],
        },  # 项目提供的可执行命令
        include_package_data=True,  # 是否包含非代码文件，如数据文件、配置文件等

--- a/magic_pdf/train_utils/__init__.py
+++ b/magic_pdf/train_utils/__init__.py
--- a/tests/test_tools/assets/cli/path/cli_test_01.pdf
+++ b/tests/test_tools/assets/cli/path/cli_test_01.pdf
--- a/tests/test_tools/assets/cli/path/cli_test_02.pdf
+++ b/tests/test_tools/assets/cli/path/cli_test_02.pdf
--- a/tests/test_tools/assets/cli/pdf/cli_test_01.pdf
+++ b/tests/test_tools/assets/cli/pdf/cli_test_01.pdf
--- a/tests/test_tools/assets/cli_dev/cli_test_01.jsonl
+++ b/tests/test_tools/assets/cli_dev/cli_test_01.jsonl
+{"file_location":"tests/test_tools/assets/cli_dev/cli_test_01.pdf","doc_layout_result":[{"layout_dets":[{"category_id":1,"poly":[882.4013061523438,169.93817138671875,1552.350341796875,169.93817138671875,1552.350341796875,625.8263549804688,882.4013061523438,625.8263549804688],"score":0.999992311000824},{"category_id":1,"poly":[882.474853515625,1450.92822265625,1551.4490966796875,1450.92822265625,1551.4490966796875,1877.5712890625,882.474853515625,1877.5712890625],"score":0.9999903440475464},{"category_id":1,"poly":[881.6513061523438,626.2058715820312,1552.1400146484375,626.2058715820312,1552.1400146484375,1450.604736328125,881.6513061523438,1450.604736328125],"score":0.9999856352806091},{"category_id":1,"poly":[149.41075134277344,232.1595001220703,819.0465087890625,232.1595001220703,819.0465087890625,625.8865356445312,149.41075134277344,625.8865356445312],"score":0.99998539686203},{"category_id":1,"poly":[149.3945770263672,1215.5172119140625,817.8850708007812,1215.5172119140625,817.8850708007812,1304.873291015625,149.3945770263672,1304.873291015625],"score":0.9999765157699585},{"category_id":1,"poly":[882.6979370117188,1880.13916015625,1552.15185546875,1880.13916015625,1552.15185546875,2031.339599609375,882.6979370117188,2031.339599609375],"score":0.9999744892120361},{"category_id":1,"poly":[148.96054077148438,743.3055419921875,818.6231689453125,743.3055419921875,818.6231689453125,1074.2369384765625,148.96054077148438,1074.2369384765625],"score":0.9999669790267944},{"category_id":1,"poly":[148.8435516357422,1791.14306640625,818.6885375976562,1791.14306640625,818.6885375976562,2030.794189453125,148.8435516357422,2030.794189453125],"score":0.9999618530273438},{"category_id":0,"poly":[150.7009735107422,684.0087890625,623.5106201171875,684.0087890625,623.5106201171875,717.03662109375,150.7009735107422,717.03662109375],"score":0.9999415278434753},{"category_id":8,"poly":[146.48068237304688,1331.6737060546875,317.2640075683594,1331.6737060546875,317.2640075683594,1400.1722412109375,146.48068237304688,1400.1722412109375],"score":0.9998958110809326},{"category_id":1,"poly":[149.42420959472656,1430.8782958984375,818.9042358398438,1430.8782958984375,818.9042358398438,1672.7386474609375,149.42420959472656,1672.7386474609375],"score":0.9998599290847778},{"category_id":1,"poly":[149.18746948242188,172.10252380371094,818.5662231445312,172.10252380371094,818.5662231445312,230.4594268798828,149.18746948242188,230.4594268798828],"score":0.9997718334197998},{"category_id":0,"poly":[149.0175018310547,1732.1090087890625,702.1005859375,1732.1090087890625,702.1005859375,1763.6046142578125,149.0175018310547,1763.6046142578125],"score":0.9997085928916931},{"category_id":2,"poly":[1519.802490234375,98.59099578857422,1551.985107421875,98.59099578857422,1551.985107421875,119.48420715332031,1519.802490234375,119.48420715332031],"score":0.9995552897453308},{"category_id":8,"poly":[146.9109649658203,1100.156494140625,544.2803344726562,1100.156494140625,544.2803344726562,1184.929443359375,146.9109649658203,1184.929443359375],"score":0.9995207786560059},{"category_id":2,"poly":[148.11611938476562,99.87767791748047,318.926025390625,99.87767791748047,318.926025390625,120.70393371582031,148.11611938476562,120.70393371582031],"score":0.999351441860199},{"category_id":9,"poly":[791.7642211914062,1130.056396484375,818.6940307617188,1130.056396484375,818.6940307617188,1161.1080322265625,791.7642211914062,1161.1080322265625],"score":0.9908884763717651},{"category_id":9,"poly":[788.37060546875,1346.8450927734375,818.5010986328125,1346.8450927734375,818.5010986328125,1377.370361328125,788.37060546875,1377.370361328125],"score":0.9873985052108765},{"category_id":14,"poly":[146,1103,543,1103,543,1184,146,1184],"score":0.94,"latex":"E\\!\\left(W\\right)\\!=\\!\\frac{E\\!\\left[H^{2}\\right]}{2E\\!\\left[H\\right]}\\!=\\!\\frac{E\\!\\left[H\\right]}{2}\\!\\!\\left(1\\!+\\!\\operatorname{CV}\\!\\left(H\\right)^{2}\\right)"},{"category_id":13,"poly":[1196,354,1278,354,1278,384,1196,384],"score":0.91,"latex":"p(1-q)"},{"category_id":13,"poly":[881,415,1020,415,1020,444,881,444],"score":0.91,"latex":"(1-p)(1-q)"},{"category_id":14,"poly":[147,1333,318,1333,318,1400,147,1400],"score":0.91,"latex":"\\mathbf{CV}\\big(H\\big)\\!=\\!\\frac{\\boldsymbol{\\upsigma}_{H}}{E\\big[H\\big]}"},{"category_id":13,"poly":[1197,657,1263,657,1263,686,1197,686],"score":0.9,"latex":"(1-p)"},{"category_id":13,"poly":[213,1217,263,1217,263,1244,213,1244],"score":0.88,"latex":"E[X]"},{"category_id":13,"poly":[214,1434,245,1434,245,1459,214,1459],"score":0.87,"latex":"\\upsigma_{H}"},{"category_id":13,"poly":[324,2002,373,2002,373,2028,324,2028],"score":0.84,"latex":"30\\%"},{"category_id":13,"poly":[1209,693,1225,693,1225,717,1209,717],"score":0.83,"latex":"p"},{"category_id":13,"poly":[990,449,1007,449,1007,474,990,474],"score":0.81,"latex":"p"},{"category_id":13,"poly":[346,1277,369,1277,369,1301,346,1301],"score":0.81,"latex":"H"},{"category_id":13,"poly":[1137,661,1154,661,1154,686,1137,686],"score":0.81,"latex":"p"},{"category_id":13,"poly":[522,1432,579,1432,579,1459,522,1459],"score":0.81,"latex":"H\\left(4\\right)"},{"category_id":13,"poly":[944,540,962,540,962,565,944,565],"score":0.8,"latex":"p"},{"category_id":13,"poly":[1444,936,1461,936,1461,961,1444,961],"score":0.79,"latex":"p"},{"category_id":13,"poly":[602,1247,624,1247,624,1270,602,1270],"score":0.78,"latex":"H"},{"category_id":13,"poly":[147,1247,167,1247,167,1271,147,1271],"score":0.77,"latex":"X"},{"category_id":13,"poly":[210,1246,282,1246,282,1274,210,1274],"score":0.77,"latex":"\\operatorname{CV}(H)"},{"category_id":13,"poly":[1346,268,1361,268,1361,292,1346,292],"score":0.76,"latex":"q"},{"category_id":13,"poly":[215,957,238,957,238,981,215,981],"score":0.74,"latex":"H"},{"category_id":13,"poly":[149,956,173,956,173,981,149,981],"score":0.63,"latex":"W"},{"category_id":13,"poly":[924,841,1016,841,1016,868,924,868],"score":0.56,"latex":"8{\\cdot}00\\;\\mathrm{a.m}"},{"category_id":13,"poly":[956,871,1032,871,1032,898,956,898],"score":0.43,"latex":"20~\\mathrm{min}"},{"category_id":13,"poly":[1082,781,1112,781,1112,808,1082,808],"score":0.41,"latex":"(l)"},{"category_id":13,"poly":[697,1821,734,1821,734,1847,697,1847],"score":0.3,"latex":"^{1\\mathrm{~h~}}"}],"page_info":{"page_no":0,"height":2200,"width":1700}}]}
\ No newline at end of file
--- a/tests/test_tools/assets/cli_dev/cli_test_01.model.json
+++ b/tests/test_tools/assets/cli_dev/cli_test_01.model.json
--- a/tests/test_tools/assets/cli_dev/cli_test_01.pdf
+++ b/tests/test_tools/assets/cli_dev/cli_test_01.pdf
--- a/tests/test_tools/assets/common/cli_test_01.pdf
+++ b/tests/test_tools/assets/common/cli_test_01.pdf
--- a/tests/test_tools/test_cli.py
+++ b/tests/test_tools/test_cli.py
+import tempfile
+import os
+import shutil
+from click.testing import CliRunner
+
+from magic_pdf.tools.cli import cli
+
+
+def test_cli_pdf():
+    # setup
+    unitest_dir = "/tmp/magic_pdf/unittest/tools"
+    filename = "cli_test_01"
+    os.makedirs(unitest_dir, exist_ok=True)
+    temp_output_dir = tempfile.mkdtemp(dir="/tmp/magic_pdf/unittest/tools")
+
+    # run
+    runner = CliRunner()
+    result = runner.invoke(
+        cli,
+        [
+            "-p",
+            "tests/test_tools/assets/cli/pdf/cli_test_01.pdf",
+            "-o",
+            temp_output_dir,
+        ],
+    )
+
+    # check
+    assert result.exit_code == 0
+
+    base_output_dir = os.path.join(temp_output_dir, "cli_test_01/auto")
+
+    r = os.stat(os.path.join(base_output_dir, f"{filename}.md"))
+    assert r.st_size > 7000
+
+    r = os.stat(os.path.join(base_output_dir, "middle.json"))
+    assert r.st_size > 200000
+
+    r = os.stat(os.path.join(base_output_dir, "model.json"))
+    assert r.st_size > 15000
+
+    r = os.stat(os.path.join(base_output_dir, "origin.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "layout.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "spans.pdf"))
+    assert r.st_size > 500000
+
+    assert os.path.exists(os.path.join(base_output_dir, "images")) is True
+    assert os.path.isdir(os.path.join(base_output_dir, "images")) is True
+    assert os.path.exists(os.path.join(base_output_dir, "content_list.json")) is False
+
+    # teardown
+    shutil.rmtree(temp_output_dir)
+
+
+def test_cli_path():
+    # setup
+    unitest_dir = "/tmp/magic_pdf/unittest/tools"
+    os.makedirs(unitest_dir, exist_ok=True)
+    temp_output_dir = tempfile.mkdtemp(dir="/tmp/magic_pdf/unittest/tools")
+
+    # run
+    runner = CliRunner()
+    result = runner.invoke(
+        cli, ["-p", "tests/test_tools/assets/cli/path", "-o", temp_output_dir]
+    )
+
+    # check
+    assert result.exit_code == 0
+
+    filename = "cli_test_01"
+    base_output_dir = os.path.join(temp_output_dir, "cli_test_01/auto")
+
+    r = os.stat(os.path.join(base_output_dir, f"{filename}.md"))
+    assert r.st_size > 7000
+
+    r = os.stat(os.path.join(base_output_dir, "middle.json"))
+    assert r.st_size > 200000
+
+    r = os.stat(os.path.join(base_output_dir, "model.json"))
+    assert r.st_size > 15000
+
+    r = os.stat(os.path.join(base_output_dir, "origin.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "layout.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "spans.pdf"))
+    assert r.st_size > 500000
+
+    assert os.path.exists(os.path.join(base_output_dir, "images")) is True
+    assert os.path.isdir(os.path.join(base_output_dir, "images")) is True
+    assert os.path.exists(os.path.join(base_output_dir, "content_list.json")) is False
+
+    base_output_dir = os.path.join(temp_output_dir, "cli_test_02/auto")
+    filename = "cli_test_02"
+
+    r = os.stat(os.path.join(base_output_dir, f"{filename}.md"))
+    assert r.st_size > 5000
+
+    r = os.stat(os.path.join(base_output_dir, "middle.json"))
+    assert r.st_size > 200000
+
+    r = os.stat(os.path.join(base_output_dir, "model.json"))
+    assert r.st_size > 15000
+
+    r = os.stat(os.path.join(base_output_dir, "origin.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "layout.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "spans.pdf"))
+    assert r.st_size > 500000
+
+    assert os.path.exists(os.path.join(base_output_dir, "images")) is True
+    assert os.path.isdir(os.path.join(base_output_dir, "images")) is True
+    assert os.path.exists(os.path.join(base_output_dir, "content_list.json")) is False
+
+    # teardown
+    shutil.rmtree(temp_output_dir)
--- a/tests/test_tools/test_cli_dev.py
+++ b/tests/test_tools/test_cli_dev.py
+import tempfile
+import os
+import shutil
+from click.testing import CliRunner
+
+from magic_pdf.tools import cli_dev
+
+
+def test_cli_pdf():
+    # setup
+    unitest_dir = "/tmp/magic_pdf/unittest/tools"
+    filename = "cli_test_01"
+    os.makedirs(unitest_dir, exist_ok=True)
+    temp_output_dir = tempfile.mkdtemp(dir="/tmp/magic_pdf/unittest/tools")
+
+    # run
+    runner = CliRunner()
+    result = runner.invoke(
+        cli_dev.cli,
+        [
+            "pdf",
+            "-p",
+            "tests/test_tools/assets/cli/pdf/cli_test_01.pdf",
+            "-j",
+            "tests/test_tools/assets/cli_dev/cli_test_01.model.json",
+            "-o",
+            temp_output_dir,
+        ],
+    )
+
+    # check
+    assert result.exit_code == 0
+
+    base_output_dir = os.path.join(temp_output_dir, "cli_test_01/auto")
+
+    r = os.stat(os.path.join(base_output_dir, "content_list.json"))
+    assert r.st_size > 5000
+
+    r = os.stat(os.path.join(base_output_dir, f"{filename}.md"))
+    assert r.st_size > 7000
+
+    r = os.stat(os.path.join(base_output_dir, "middle.json"))
+    assert r.st_size > 200000
+
+    r = os.stat(os.path.join(base_output_dir, "model.json"))
+    assert r.st_size > 15000
+
+    r = os.stat(os.path.join(base_output_dir, "origin.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "layout.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "spans.pdf"))
+    assert r.st_size > 500000
+
+    assert os.path.exists(os.path.join(base_output_dir, "images")) is True
+    assert os.path.isdir(os.path.join(base_output_dir, "images")) is True
+
+    # teardown
+    shutil.rmtree(temp_output_dir)
+
+
+def test_cli_jsonl():
+    # setup
+    unitest_dir = "/tmp/magic_pdf/unittest/tools"
+    filename = "cli_test_01"
+    os.makedirs(unitest_dir, exist_ok=True)
+    temp_output_dir = tempfile.mkdtemp(dir="/tmp/magic_pdf/unittest/tools")
+
+    def mock_read_s3_path(s3path):
+        with open(s3path, "rb") as f:
+            return f.read()
+
+    cli_dev.read_s3_path = mock_read_s3_path # mock
+
+    # run
+    runner = CliRunner()
+    result = runner.invoke(
+        cli_dev.cli,
+        [
+            "jsonl",
+            "-j",
+            "tests/test_tools/assets/cli_dev/cli_test_01.jsonl",
+            "-o",
+            temp_output_dir,
+        ],
+    )
+
+    # check
+    assert result.exit_code == 0
+
+    base_output_dir = os.path.join(temp_output_dir, "cli_test_01/auto")
+
+    r = os.stat(os.path.join(base_output_dir, "content_list.json"))
+    assert r.st_size > 5000
+
+    r = os.stat(os.path.join(base_output_dir, f"{filename}.md"))
+    assert r.st_size > 7000
+
+    r = os.stat(os.path.join(base_output_dir, "middle.json"))
+    assert r.st_size > 200000
+
+    r = os.stat(os.path.join(base_output_dir, "model.json"))
+    assert r.st_size > 15000
+
+    r = os.stat(os.path.join(base_output_dir, "origin.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "layout.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "spans.pdf"))
+    assert r.st_size > 500000
+
+    assert os.path.exists(os.path.join(base_output_dir, "images")) is True
+    assert os.path.isdir(os.path.join(base_output_dir, "images")) is True
+
+    # teardown
+    shutil.rmtree(temp_output_dir)
--- a/tests/test_tools/test_common.py
+++ b/tests/test_tools/test_common.py
+import tempfile
+import os
+import shutil
+
+import pytest
+
+from magic_pdf.tools.common import do_parse
+
+
+@pytest.mark.parametrize("method", ["auto", "txt", "ocr"])
+def test_common_do_parse(method):
+    # setup
+    unitest_dir = "/tmp/magic_pdf/unittest/tools"
+    filename = "fake"
+    os.makedirs(unitest_dir, exist_ok=True)
+
+    temp_output_dir = tempfile.mkdtemp(dir="/tmp/magic_pdf/unittest/tools")
+
+    # run
+    with open("tests/test_tools/assets/common/cli_test_01.pdf", "rb") as f:
+        bits = f.read()
+    do_parse(temp_output_dir, filename, bits, [], method, f_dump_content_list=True)
+
+    # check
+    base_output_dir = os.path.join(temp_output_dir, f"fake/{method}")
+
+    r = os.stat(os.path.join(base_output_dir, "content_list.json"))
+    assert r.st_size > 5000
+
+    r = os.stat(os.path.join(base_output_dir, f"{filename}.md"))
+    assert r.st_size > 7000
+
+    r = os.stat(os.path.join(base_output_dir, "middle.json"))
+    assert r.st_size > 200000
+
+    r = os.stat(os.path.join(base_output_dir, "model.json"))
+    assert r.st_size > 15000
+
+    r = os.stat(os.path.join(base_output_dir, "origin.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "layout.pdf"))
+    assert r.st_size > 500000
+
+    r = os.stat(os.path.join(base_output_dir, "spans.pdf"))
+    assert r.st_size > 500000
+
+    os.path.exists(os.path.join(base_output_dir, "images"))
+    os.path.isdir(os.path.join(base_output_dir, "images"))
+
+    # teardown
+    shutil.rmtree(temp_output_dir)