ToDo: Project Checkpoints

[KryptixOne]

Project Checkpoints:

---

• Take the following as inputs from the User:
  • Sequence of Reference Images or a Single Image
    • Function for Acquisition and Transformation developed
    • Integrate into SD pipeline
  • Desired Prompt
    • Integrated into SD Pipeline
  • Desired Negative Prompt. Or This can be predetermined by us.
    • Integrated into SD Pipeline

---

• Build Base In-Painting Img2Img and DreamBooth Pipeline For SD
  • Img2Img Integration
  • DreamBooth Integration *Not Required
    • Temporary Hold on Dreambooth Integration. Training is too resource intensive and it takes too long to train
  • In-Painting Integration (Don't worry about mask creation yet)

---

• Build workflow that ultilizes each pipeline and produces output images given an input
  Note that the goal here blends with goal number 4
  • Inpainting Method Workflow Creation
    • Acquire Reference Image of object
    • Create Mask for Img2Img Inpainting
      • Implement Segment Anything Model (Note that text prompt input not public
      • Implement CLIP for prompt-to-mask similarity scoring to enable text prompt
      • Fill in holes in mask detection
        • Small hole fill in using morphology
        • Contour-level fill
      • Identify first 5 most relevant masks
    • Generate Img
      • Generated images affecting non-masked areas.
  • Build secondary Mask for Generated Image (on borders)
  • Generate Img again using previous generated img and new mask.
  • Save image.

---

• Nice-To-Haves:
  • LoRA integration (from .safetensors if using publicly available LoRAs)
  • Color Corrections
  • First Pass Auto-Beautify Filter (May include auto correction)
  • Multiple-Beauty Filters on first pass (allow selection prior to second pass)
  • When image is just the Shirt without Human structural Information. Superimpose on random model/manikin

---

• Identify area of interest based on reference images.
  • Create masks with segmentation net.
    • "Segment Anything" Paper
    • Look into fast Segment anything. This has a free commercial use license
  • Define Inpainting area
  • Define Border of Inpainting area for Second pass to smooth border artifacts
    * Not required if Segmentation Mask is of high Quality
  • If using DreamBooth-like methodology, Not required

---

• Hyperparameter Tuning

---

• Incorporate Visual Similarity Metric.
  • Metric will rate how "similar" the generated object is to reference.
  • Use segmentation net to identify AoI
  • Need to determine this metric still. As variation, pose, angle, lighting, etc. should not negatively affect
    the Metric but visual distortions to the reference should

---

• Incorporate Image Filtering Based on Similarity Metric
  • Remove Images that don't achieve a certain threshold.
  • For Successful Images, Log Hyperparameters, seed, and reference Images (useful for future training)
  • Return X number of Generated Images
    --
• Build a WebUI that Allows for independent user usage
  • Try different frameworks
    • Streamlit
    • Gradio
    • Flask
  • Determine which framework best suits your needs
  • Build sections using desired framework
    • Homepage
    • About Us
    • Examples
    • Demo
      • Design layout
      • Create UI
      • Build Functionality with ML Model
      • Incorporate Cloud-Cluster Compatibility
      • Create Demo Video
    • Contact