Project-Specific Learnings¶

This file accumulates knowledge from each conversion project to avoid repeating mistakes.

Important: When adding learnings from a new project, extract only the essential insights and generalizable lessons. Focus on:

• Key challenges unique to the project
• Solutions that can be applied to future conversions
• Patterns or anti-patterns discovered
• Configuration values that differ from expectations

Avoid verbose descriptions. Keep each entry concise and actionable.

MVANet (Dichotomous Image Segmentation) - 2026-01-12¶

Architecture: Multi-view aggregation with SwinB backbone

Key challenges & solutions:

1. MCRM pool_ratios mismatch

2. Original implementation used [2, 4, 8] in actual model

3. Config had wrong default [4, 8, 16]

4. Lesson: Always verify default values match pretrained weights by tracing through actual model instantiation

5. Multi-view architecture handling

6. Model uses 4 local patches + 1 global view

7. Batch dimension needs careful handling: batch_size * 5

8. Lesson: Document multi-view assumptions clearly in config

9. Preprocessing exact match

10. torchvision.transforms vs numpy+torch gave identical results

11. Critical: use same resize method (BILINEAR) and exact same mean/std

12. Lesson: Preprocessing can be implemented differently as long as output is identical

13. Double-resizing issue

14. Tests showed 0.02% difference when using different preprocessed tensors

15. Using same tensor → bitwise identical outputs

16. Lesson: Always use SAME preprocessed tensor when comparing models for fair comparison

17. Backbone output channels

18. SwinB specific: [128, 128, 256, 512, 1024]

19. Made configurable for potential future backbone changes

20. Lesson: Make backbone-specific constants configurable even if unlikely to change

21. Positional encoding buffer handling

22. dim_t buffer needs explicit device movement

23. Not automatically moved by .to(device)

24. Lesson: Check all submodules for buffers that need manual device placement

25. Auto* class registration required for Hub

26. Must call register_for_auto_class() on each class before pushing to Hub

27. Required for from_pretrained() to work correctly after uploading
28. Lesson: Always register custom models using register_for_auto_class():

    ModelConfig.register_for_auto_class()
    ModelForTask.register_for_auto_class("AutoModel")
    ImageProcessor.register_for_auto_class("AutoImageProcessor")
    

29. Best practice: Add registration in __init__.py so it happens on module import

30. Image size order in post-processing

31. post_process_semantic_segmentation() expects target_sizes as (width, height)

32. PIL Image.size returns (width, height) ✓
33. PIL Image.width and Image.height properties exist separately

34. Lesson: When using target_sizes, always use (width, height) order:

    # Correct
    target_sizes = [(img.width, img.height) for img in images]
    # or
    w, h = img.size
    target_sizes = [(w, h)]
    
    # Wrong - causes width/height swap
    target_sizes = [(img.height, img.width)]
    

[Project Name] ([Task Type]) - [Date]¶

Architecture: [Brief architecture description]

Key challenges & solutions:

1. [Challenge name]

2. [Description of challenge]

3. [How it was solved]

4. Lesson: [Generalizable insight for future conversions]

5. [Challenge name]

6. [Description]
7. [Solution]
8. Lesson: [Insight]

[Add more entries as you gain experience with different model types]