multimodal-maesto is out 🔥 🔥 🔥

🚀 Added

• SegmentAnythingMarkGenerator allowing the generation of segmentation marks.
• MarkVisualizer allowing to visualize the generated marks.
• prompt_image allowing for convenient GPT-4 Vision API querying.
• 🤗 Hugging Face Set-of-Mark space.

>>> import cv2
>>> import torch
>>> import multimodalmaesto as mm

>>> image = cv2.imread("...")

>>> generator = mm.SegmentAnythingMarkGenerator()
>>> visualizer = mm.MarkVisualizer()

>>> marks = generator.generate(image=image)
>>> marks = mm.refine_marks(marks=marks)

>>> image_prompt = visualizer.visualize(image=image, marks=marks)
>>> text_prompt = "Find dog."

>>> response = mm.prompt_image(api_key=api_key, image=image_prompt, prompt=text_prompt)
>>> response

"The dog is prominently featured in the center of the image with the label [9]."

>>> masks = mm.extract_relevant_masks(text=response, detections=refined_marks)

{'6': array([
    [False, False, False, ..., False, False, False],
    [False, False, False, ..., False, False, False],
    [False, False, False, ..., False, False, False],
    ...,
    [ True,  True,  True, ..., False, False, False],
    [ True,  True,  True, ..., False, False, False],
    [ True,  True,  True, ..., False, False, False]])
}

🏆 Contributors

@SkalskiP @deependujha