wangshuxun/zotero-kb
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
9ba1668a66
zotero-kb/docs/superpowers/plans/2026-04-15-import-card-two-step.md
Saberlve cc1960e9a9 docs: add two-step import+card generation implementation plan 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-15 17:04:08 +08:00

663 lines
24 KiB
Markdown
Raw Blame History

# Two-Step Import + Card Generation Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Split the current "import文献 + immediately generate cards" flow into: (1) import items to project, check global cards.json for existing cards, (2) manual batch select + generate cards.
**Architecture:** Backend: two new API endpoints (`import-state`, `cards/generate`). `imports/item-keys` no longer generates cards — it only registers items. `CardBuilder.build_or_update()` already computes `source_hash` and skips LLM if hash unchanged (dedup). Frontend: import window two-column layout with pending/done visual states and batch generate button with real-time progress.
**Tech Stack:** Python/FastAPI, vanilla JS, HTML/CSS.
---
## Task 1: Backend — `GET /api/projects/{project_id}/import-state`
**Files:**
- Modify: `src/zotero_kb/api.py` (add endpoint)
- Test: `tests/test_api.py`
- [ ] **Step 1: Write the failing test**
In `tests/test_api.py`, add test `test_import_state_returns_pending_and_done`:
```python
def test_import_state_returns_pending_and_done(tmp_path, monkeypatch):
from zotero_kb.api import create_app
from zotero_kb.config import AppConfig
from zotero_kb.llm import DeterministicCardClient
# Setup: create project with 2 items, one has card
workspace = tmp_path / "workspace"
items_index = workspace / "library" / "index" / "items.json"
cards_index = workspace / "library" / "index" / "cards.json"
items_index.parent.mkdir(parents=True)
cards_index.parent.mkdir(parents=True)
items_index.write_text(json.dumps({
"KEY1": {"item_key": "KEY1", "title": "Paper A"},
"KEY2": {"item_key": "KEY2", "title": "Paper B"},
}))
cards_index.write_text(json.dumps({
"KEY1": {"item_key": "KEY1", "title": "Paper A"} # KEY1 has card, KEY2 does not
}))
project_dir = workspace / "projects" / "proj1"
project_dir.mkdir(parents=True)
(project_dir / "selected-items.json").write_text(json.dumps(["KEY1", "KEY2"]))
config = AppConfig(workspace_dir=workspace, zotero_data_dir=tmp_path / "zotero")
app = create_app(config, llm_client=DeterministicCardClient())
response = client.get("/api/projects/proj1/import-state")
assert response.status_code == 200
data = response.json()
assert data["pending_count"] == 1
assert data["done_count"] == 1
items_by_key = {it["item_key"]: it for it in data["items"]}
assert items_by_key["KEY1"]["card_status"] == "done"
assert items_by_key["KEY2"]["card_status"] == "pending"
```
- [ ] **Step 2: Run test to verify it fails**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_api.py::test_import_state_returns_pending_and_done -v`
Expected: FAIL — endpoint not defined
- [ ] **Step 3: Implement the endpoint**
Add to `api.py`:
```python
@app.get("/api/projects/{project_id}/import-state")
def get_import_state(project_id: str) -> dict[str, object]:
project_dir = config.workspace_dir / "projects" / project_id
project_file = project_dir / "project.json"
if not project_file.exists():
raise HTTPException(status_code=404, detail="Project not found")
selected_items = project_service._read_selected_items(project_id)
cards_index = projects._read_json(config.workspace_dir / "library" / "index" / "cards.json")
items_index = projects._read_json(config.workspace_dir / "library" / "index" / "items.json")
items = []
pending_count = 0
done_count = 0
for item_key in selected_items:
item_data = items_index.get(item_key, {})
card_status = "done" if item_key in cards_index else "pending"
if card_status == "done":
done_count += 1
else:
pending_count += 1
items.append({
"item_key": item_key,
"title": item_data.get("title", ""),
"creators": item_data.get("creators", []),
"year": item_data.get("year"),
"item_type": item_data.get("item_type", ""),
"card_status": card_status,
})
return {
"project_id": project_id,
"items": items,
"pending_count": pending_count,
"done_count": done_count,
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_api.py::test_import_state_returns_pending_and_done -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/zotero_kb/api.py tests/test_api.py
git commit -m "feat: add GET /api/projects/{id}/import-state endpoint
Returns pending/done status for each item in a project by checking
selected-items.json against cards.json.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
"
```
---
## Task 2: Backend — Modify `POST /api/projects/{project_id}/imports/item-keys` to not generate cards
**Files:**
- Modify: `src/zotero_kb/api.py` (lines ~127-155, the `import_item_keys` endpoint)
- Test: `tests/test_api.py`
- [ ] **Step 1: Write the failing test**
Add `test_import_item_keys_does_not_generate_cards`:
```python
def test_import_item_keys_does_not_generate_cards(tmp_path, monkeypatch):
from zotero_kb.api import create_app
from zotero_kb.config import AppConfig
from zotero_kb.llm import DeterministicCardClient
workspace = tmp_path / "workspace"
items_index = workspace / "library" / "index" / "items.json"
items_index.parent.mkdir(parents=True)
items_index.write_text(json.dumps({}))
project_dir = workspace / "projects" / "proj1"
project_dir.mkdir(parents=True)
(project_dir / "project.json").write_text(json.dumps({"project_id": "proj1", "name": "Test"}))
config = AppConfig(workspace_dir=workspace, zotero_data_dir=tmp_path / "zotero")
app = create_app(config, llm_client=DeterministicCardClient())
# Mock reader to return fake items
class FakeReader:
def read_items(self, keys):
return [ZoteroItemRecord(item_key=k, title=f"Paper {k}", creators=[], year=2024,
item_type="journalArticle", abstract="", tags=[], collection_paths=[],
notes=[], attachment_texts=[]) for k in keys]
monkeypatch.setattr("zotero_kb.api.ZoteroReader", lambda *a, **k: FakeReader())
response = client.post("/api/projects/proj1/imports/item-keys", json={"item_keys": ["KEY1"]})
assert response.status_code == 200
data = response.json()
assert data["imported_item_keys"] == ["KEY1"]
# Verify cards.json was NOT created (no card generation)
cards_index = workspace / "library" / "index" / "cards.json"
assert not cards_index.exists()
```
- [ ] **Step 2: Run test to verify it fails**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_api.py::test_import_item_keys_does_not_generate_cards -v`
Expected: FAIL — currently it calls `CardBuilder.build_or_update()` which creates cards
- [ ] **Step 3: Modify `import_item_keys` to not generate cards**
Change the `import_item_keys` endpoint body from:
```python
builder = CardBuilder(config.workspace_dir, resolved_client)
items = reader.read_items(item_keys)
imported_keys = []
for item in items:
builder.build_or_update(item) # REMOVE THIS
imported_keys.append(item.item_key)
```
to:
```python
items = reader.read_items(item_keys)
imported_keys = [item.item_key for item in items]
# NOTE: card generation happens later via explicit /cards/generate call
# Ensure items are registered in items.json for later reference
items_index = projects._read_json(config.workspace_dir / "library" / "index" / "items.json")
for item in items:
items_index[item.item_key] = {
"item_key": item.item_key,
"title": item.title,
"creators": item.creators,
"year": item.year,
"item_type": item.item_type,
"abstract": item.abstract,
"tags": item.tags,
"collection_paths": item.collection_paths,
}
projects._write_json(config.workspace_dir / "library" / "index" / "items.json", items_index)
```
Also remove the LLM client creation since it's no longer needed for this endpoint.
- [ ] **Step 4: Run test to verify it passes**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_api.py::test_import_item_keys_does_not_generate_cards -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/zotero_kb/api.py tests/test_api.py
git commit -m "refactor: import_item_keys no longer generates cards immediately
Items are registered in items.json but cards are generated later via
explicit POST /projects/{id}/cards/generate call.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
"
```
---
## Task 3: Backend — `POST /api/projects/{project_id}/cards/generate`
**Files:**
- Modify: `src/zotero_kb/api.py` (add endpoint)
- Modify: `src/zotero_kb/projects.py` (add `add_items` helper)
- Test: `tests/test_api.py`
- [ ] **Step 1: Write the failing test**
Add `test_cards_generate_creates_cards_for_pending_items`:
```python
def test_cards_generate_creates_cards_for_pending_items(tmp_path, monkeypatch):
from zotero_kb.api import create_app
from zotero_kb.config import AppConfig
from zotero_kb.llm import DeterministicCardClient
workspace = tmp_path / "workspace"
items_index = workspace / "library" / "index" / "items.json"
cards_index = workspace / "library" / "index" / "cards.json"
items_index.parent.mkdir(parents=True)
cards_index.parent.mkdir(parents=True)
items_index.write_text(json.dumps({
"KEY1": {"item_key": "KEY1", "title": "Paper A", "creators": ["Author A"], "year": 2024,
"item_type": "journalArticle", "abstract": "Abstract A", "tags": [], "collection_paths": []},
}))
cards_index.write_text(json.dumps({})) # no existing cards
project_dir = workspace / "projects" / "proj1"
project_dir.mkdir(parents=True)
(project_dir / "project.json").write_text(json.dumps({"project_id": "proj1", "name": "Test",
"llm": {"provider": "deterministic", "model": "deterministic"}}))
(project_dir / "selected-items.json").write_text(json.dumps(["KEY1"]))
config = AppConfig(workspace_dir=workspace, zotero_data_dir=tmp_path / "zotero")
app = create_app(config, llm_client=DeterministicCardClient())
response = client.post("/api/projects/proj1/cards/generate", json={"item_keys": ["KEY1"]})
assert response.status_code == 200
data = response.json()
assert data["generated"] == ["KEY1"]
assert data["failed"] == []
assert data["items"][0]["card_status"] == "done"
# Verify cards.json now has the card
cards = json.loads(cards_index.read_text())
assert "KEY1" in cards
```
- [ ] **Step 2: Run test to verify it fails**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_api.py::test_cards_generate_creates_cards_for_pending_items -v`
Expected: FAIL — endpoint not defined
- [ ] **Step 3: Implement the endpoint**
Add to `api.py`:
```python
class GenerateCardsRequest(BaseModel):
item_keys: list[str] = Field(min_length=1)
@app.post("/api/projects/{project_id}/cards/generate")
def generate_cards(project_id: str, payload: GenerateCardsRequest | dict[str, object]) -> dict[str, object]:
project_file = config.workspace_dir / "projects" / project_id / "project.json"
if not project_file.exists():
raise HTTPException(status_code=404, detail="Project not found")
if isinstance(payload, dict):
item_keys = [str(value) for value in payload.get("item_keys", [])]
else:
item_keys = [str(value) for value in payload.item_keys]
project_payload = _read_json(project_file)
llm_payload = project_payload.get("llm", {})
try:
resolved_client = llm_client or create_card_generation_client(
str(llm_payload.get("provider", "deterministic")),
str(llm_payload.get("model", "deterministic")),
)
except ValueError as exc:
raise HTTPException(status_code=400, detail=str(exc)) from exc
builder = CardBuilder(config.workspace_dir, resolved_client)
reader = ZoteroReader(config.zotero_data_dir)
items = reader.read_items(item_keys)
generated = []
failed = []
result_items = []
for item in items:
try:
result = builder.build_or_update(item)
generated.append(item.item_key)
card_status = "done"
except Exception as exc:
failed.append(item.item_key)
card_status = "failed"
result_items.append({
"item_key": item.item_key,
"title": item.title,
"creators": item.creators,
"year": item.year,
"item_type": item.item_type,
"card_status": card_status,
})
project_service.add_items(project_id, generated)
return {
"project_id": project_id,
"generated": generated,
"failed": failed,
"items": result_items,
}
```
- [ ] **Step 4: Run test to verify it passes**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_api.py::test_cards_generate_creates_cards_for_pending_items -v`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/zotero_kb/api.py tests/test_api.py
git commit -m "feat: add POST /api/projects/{id}/cards/generate endpoint
Generates cards for specified item_keys via CardBuilder. Returns
generated/failed lists and updated item states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
"
```
---
## Task 4: Frontend — Import Window Two-Column Layout
**Files:**
- Modify: `src/zotero_kb/templates/index.html` (import window HTML + JS)
This is the most complex task. Changes span:
1. Add import-state API call function
2. Add cards/generate API call function
3. Replace current import window content with two-column layout
4. Add pending/done visual states and checkbox logic
5. Add batch generate button with progress feedback
**Layout structure (HTML, inside `.window-frame.import-window`):**
```html
<div class="import-split">
<div class="import-left">
<div class="import-list-header">
<span>Pending (<span id="pending-count">0</span>)</span>
<button type="button" id="select-all-pending" class="link-btn">Select All</button>
</div>
<div class="import-list" id="import-items-list">
<!-- populated by JS -->
</div>
<div class="import-list-footer">
Done (<span id="done-count">0</span>)
</div>
</div>
<div class="import-right">
<div id="item-preview">
<p class="hint">Click an item to preview</p>
</div>
</div>
</div>
<div class="import-actions">
<button type="button" id="generate-cards-btn" class="primary" disabled>生成选中卡片 (0)</button>
</div>
```
**CSS additions:**
```css
.import-split { display: grid; grid-template-columns: 40% 60%; height: 100%; gap: 1px; background: var(--line); }
.import-left, .import-right { background: var(--surface); overflow-y: auto; padding: 1rem; }
.import-list-header { display: flex; justify-content: space-between; align-items: center; margin-bottom: 0.75rem; font-weight: 600; }
.import-list { display: flex; flex-direction: column; gap: 0.25rem; }
.import-item { display: flex; align-items: flex-start; gap: 0.5rem; padding: 0.5rem; border-radius: 6px; cursor: pointer; }
.import-item.pending { background: var(--panel); }
.import-item.done { background: transparent; opacity: 0.7; }
.import-item.selected { outline: 2px solid var(--accent, #4a90d9); }
.import-item input[type="checkbox"] { margin-top: 0.25rem; }
.import-item.done input[type="checkbox"] { display: none; }
.import-item .item-title { font-size: 0.875rem; font-weight: 500; }
.import-item .item-meta { font-size: 0.75rem; color: var(--ink-dim, #666); }
.import-list-footer { margin-top: 0.75rem; font-weight: 600; font-size: 0.875rem; }
.import-actions { padding: 0.75rem 1rem; border-top: 1px solid var(--line); display: flex; justify-content: flex-end; }
#generate-cards-btn:disabled { opacity: 0.5; cursor: not-allowed; }
#generate-cards-btn .progress { font-size: 0.875rem; }
.link-btn { background: none; border: none; color: var(--accent, #4a90d9); cursor: pointer; font-size: 0.8rem; }
#item-preview { }
#item-preview .preview-title { font-size: 1rem; font-weight: 600; margin-bottom: 0.5rem; }
#item-preview .preview-meta { font-size: 0.8rem; color: var(--ink-dim); margin-bottom: 0.75rem; }
#item-preview .preview-abstract { font-size: 0.85rem; line-height: 1.5; }
.hint { color: var(--ink-dim); font-size: 0.875rem; }
```
**JS additions (state + functions):**
```javascript
// State
state.importItems = []; // items from import-state
state.selectedPendingKeys = new Set();
state.isGenerating = false;
state.generatingProgress = { current: 0, total: 0 };
// Functions
async function loadImportState() {
const resp = await fetch(`/api/projects/${state.currentProjectId}/import-state`);
const data = await resp.json();
state.importItems = data.items;
renderImportItemList();
updateImportCounts(data.pending_count, data.done_count);
}
function renderImportItemList() {
const list = document.getElementById('import-items-list');
list.innerHTML = '';
state.importItems.forEach(item => {
const isDone = item.card_status === 'done';
const div = document.createElement('div');
div.className = `import-item ${item.card_status}`;
div.dataset.key = item.item_key;
div.innerHTML = `
<input type="checkbox" ${isDone ? 'disabled' : ''} data-key="${item.item_key}">
<div>
<div class="item-title">${escapeHtml(item.title)}</div>
<div class="item-meta">${item.creators.join(', ')} · ${item.year || 'n.d.'}</div>
</div>
`;
if (!isDone) {
div.querySelector('input').addEventListener('change', (e) => {
if (e.target.checked) state.selectedPendingKeys.add(item.item_key);
else state.selectedPendingKeys.delete(item.item_key);
updateGenerateButton();
div.classList.toggle('selected', e.target.checked);
});
div.addEventListener('click', (e) => { if (e.target.tagName !== 'INPUT') showItemPreview(item); });
}
div.addEventListener('click', (e) => { if (e.target.tagName !== 'INPUT') showItemPreview(item); });
list.appendChild(div);
});
}
function showItemPreview(item) {
document.getElementById('item-preview').innerHTML = `
<div class="preview-title">${escapeHtml(item.title)}</div>
<div class="preview-meta">${item.creators.join(', ')} · ${item.year || 'n.d.'} · ${item.item_type}</div>
<div class="preview-abstract">${escapeHtml(item.abstract || 'No abstract available.')}</div>
`;
}
function updateGenerateButton() {
const btn = document.getElementById('generate-cards-btn');
const count = state.selectedPendingKeys.size;
btn.disabled = count === 0 || state.isGenerating;
btn.textContent = state.isGenerating
? `生成中 ${state.generatingProgress.current}/${state.generatingProgress.total}`
: `生成选中卡片 (${count})`;
}
async function generateSelectedCards() {
const keys = Array.from(state.selectedPendingKeys);
if (!keys.length) return;
state.isGenerating = true;
state.generatingProgress = { current: 0, total: keys.length };
updateGenerateButton();
const btn = document.getElementById('generate-cards-btn');
try {
const resp = await fetch(`/api/projects/${state.currentProjectId}/cards/generate`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ item_keys: keys }),
});
const data = await resp.json();
// Update item states
data.items.forEach(updated => {
const item = state.importItems.find(i => i.item_key === updated.item_key);
if (item) item.card_status = updated.card_status;
});
state.selectedPendingKeys.clear();
renderImportItemList();
const counts = {
pending: state.importItems.filter(i => i.card_status === 'pending').length,
done: state.importItems.filter(i => i.card_status === 'done').length,
};
updateImportCounts(counts.pending, counts.done);
} finally {
state.isGenerating = false;
updateGenerateButton();
}
}
function updateImportCounts(pending, done) {
document.getElementById('pending-count').textContent = pending;
document.getElementById('done-count').textContent = done;
}
```
**Wiring:**
- In `openImportWindow()`: call `loadImportState()`
- In `closeImportWindow()`: clear `state.importItems` and `state.selectedPendingKeys`
- `document.getElementById('select-all-pending').addEventListener('click', ...)` — select all pending checkboxes
- `document.getElementById('generate-cards-btn').addEventListener('click', generateSelectedCards)`
- [ ] **Step 1: Write the failing test**
Add to `tests/test_ui.py`:
```python
def test_index_import_window_has_two_column_layout():
html = (Path(__file__).parent.parent / "src" / "zotero_kb" / "templates" / "index.html").read_text()
assert 'class="import-split"' in html
assert 'class="import-left"' in html
assert 'class="import-right"' in html
assert 'id="generate-cards-btn"' in html
assert 'id="import-items-list"' in html
assert 'id="pending-count"' in html
assert 'id="done-count"' in html
def test_index_has_import_state_api_call():
html = (Path(__file__).parent.parent / "src" / "zotero_kb" / "templates" / "index.html").read_text()
assert '/api/projects/' in html and 'import-state' in html
assert '/api/projects/' in html and 'cards/generate' in html
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_ui.py -v -k "two_column or import_state"`
Expected: FAIL — new elements don't exist yet
- [ ] **Step 3: Implement the two-column layout**
Replace the current `.window-body` content inside `.import-window` with the two-column HTML structure above, add all CSS to the `<style>` block, and add all JS functions to the `<script>` section. Wire up the event listeners.
- [ ] **Step 4: Run tests to verify they pass**
Run: `UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/test_ui.py -v -k "two_column or import_state"`
Expected: PASS
- [ ] **Step 5: Commit**
```bash
git add src/zotero_kb/templates/index.html tests/test_ui.py
git commit -m "feat: add two-column import window layout with batch generate
Import window now shows pending/done items in left column, preview in
right column. 'Generate Selected Cards' button triggers batch
generation with real-time progress.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
"
```
---
## Task 5: End-to-End Verification
**Files:**
- None (manual + test verification)
- [ ] **Step 1: Start the app and verify the flow**
```bash
UV_CACHE_DIR=/tmp/uv-cache uv run python main.py
```
1. Open http://localhost:8000
2. Create/open a project
3. Open import window — verify it shows pending items (empty cards list initially)
4. Select items → click "生成选中卡片"
5. Verify items move from pending to done after generation
- [ ] **Step 2: Run full test suite**
```bash
UV_CACHE_DIR=/tmp/uv-cache uv run pytest tests/ -v
```
Expected: All 29+ tests pass
- [ ] **Step 3: Commit**
```bash
git add -A && git commit -m "test: add UI tests for two-column import layout
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
"
```
---
## Self-Review Checklist
1. **Spec coverage:** Each requirement from `2026-04-15-import-card-two-step-design.md` is addressed:
- `GET /import-state` → Task 1
- `POST /cards/generate` → Task 3
- `import_item_keys` no longer generates → Task 2
- Two-column layout + batch generate → Task 4
- Real-time progress → Task 4 (JS progress counter in button)
- Deduplication via source_hash → Already exists in `CardBuilder.build_or_update()`
2. **Placeholder scan:** No "TBD", "TODO", or vague steps found.
3. **Type consistency:** All method names match across tasks:
- `project_service._read_selected_items()` — used in Task 1, defined in `projects.py`
- `CardBuilder.build_or_update()` — called in Task 3, already exists in `cards.py`
- `ZoteroReader.read_items()` — used in Tasks 2 and 3, already exists
- `GenerateCardsRequest` model — defined in Task 3, used in API endpoint
Reference in New Issue View Git Blame Copy Permalink