UniSAFE: Multimodal Image‑Text Safety Dataset (Trending)
UniSAFE, authored by segyulee, is a multimodal dataset that pairs images with textual instructions and metadata describing unsafe triggers, target outcomes, and scenario types. It is stored in optimized Parquet files and can be accessed via the Hugging Face datasets library as well as Dask, Polars, and mlcroissant, making it suitable for scalable data processing.
The dataset comprises two primary splits: an **image** split with 3,124 examples (≈3.4 GB) and a **text** split with 3,678 examples (≈1.9 GB), for a total size of about 5.4 GB. Each record includes fields such as `id`, `category`, `subcategory`, `unsafe_trigger`, `target`, `scenario_type`, `instruction`, `input_image`, `input_image_b`, and a list of dialogue `turns`. The presence of both image and text modalities, together with safety‑oriented annotations, positions UniSAFE as a resource for evaluating and improving the safety of multimodal AI systems.
Because the data is formatted as Parquet and optimized for column‑arrest processing, it can be efficiently loaded and queried on large‑scale compute clusters, particularly in the US region where it is hosted. Researchers and developers can leverage UniSAFE to benchmark content‑filtering models, train safe‑response generators, or build retrieval systems that surface relevant safety scenarios.
Project Ideas
- Fine‑tune a multimodal safety classifier that flags images and instructions containing unsafe triggers.
- Create a dialogue system that generates safe responses by conditioning on the `instruction` and `turns` fields.
- Build a retrieval engine that returns similar unsafe scenarios based on `category` and `subcategory` for safety testing.
- Analyze the distribution of `scenario_type` across image and text splits to identify gaps in safety coverage.
- Develop a benchmark suite that measures how well language models avoid the `target` outcomes when given the `unsafe_trigger` prompts.