Text-IF: Leveraging Semantic Text Guidance for Degradation-Aware and
Interactive Image Fusion
Xunpeng Yi, Han Xu, Hao Zhang, Linfeng Tang, Jiayi Ma
*
Electronic Information School, Wuhan University, Wuhan 430072, China
{yixunpeng, xu han}@whu.edu.cn, {zhpersonalbox, linfeng0419, jyma2010}@gmail.com
Abstract
Image fusion aims to combine information from differ-
ent source images to create a comprehensively representa-
tive image. Existing fusion methods are typically helpless in
dealing with degradations in low-quality source images and
non-interactive to multiple subjective and objective needs.
To solve them, we introduce a novel approach that leverages
semantic text guidance image fusion model for degradation-
aware and interactive image fusion task, termed as Text-
IF. It innovatively extends the classical image fusion to the
text guided image fusion along with the ability to harmo-
niously address the degradation and interaction issues dur-
ing fusion. Through the text semantic encoder and semantic
interaction fusion decoder, Text-IF is accessible to the all-
in-one infrared and visible image degradation-aware pro-
cessing and the interactive flexible fusion outcomes. In this
way, Text-IF achieves not only multi-modal image fusion,
but also multi-modal information fusion. Extensive exper-
iments prove that our proposed text guided image fusion
strategy has obvious advantages over SOTA methods in the
image fusion performance and degradation treatment. The
code is available at https://github.com/XunpengYi/Text-IF.
1. Introduction
Image fusion is a prominent field within the domain of dig-
ital image processing [15, 27, 35]. Single-modal images
can only capture partial representation of the scene. Multi-
modal images allow for the effective acquisition of more
comprehensive representation. As an important represen-
tative, visible images provide the reflectance-based visual
information, akin to human vision. Infrared images provide
thermal radiation-based information, more valuable for de-
tecting thermal targets and observing nighttime activities.
The infrared and visible image fusion focuses on fusing the
complementary information of infrared and visible images,
yielding high-quality fusion images [18–20, 28, 38, 39, 43].
*
Corresponding author
low-quality
fusion image
low-quality infrared
and visible images
Fusion
Network
predefined
fusion loss
(a) Simple fusion approach
(b) Separated approach
fusion image
low-quality infrared
and visible images
SOTA Image
Restoration Model
user-required
fusion image
Text-Image Fusion
Network (all in one)
low-quality infrared
and visible images
(c) Proposed text guided image fusion approach
Text
semantic guidance
Fusion
Network
not well-done,
tedious and
non-interactive
helpless
and
non-interactive
great and interactive!
complex scenes,
degradation
Figure 1. Fusion approaches for complex scenes with degrada-
tions. (a) simple fusion approach: treating image fusion with
predefined fusion loss and not applicable to complex scenes with
degradations. (b) separated approach: requiring frequent restora-
tion methods switching according to the type of degradations,
which is troublesome and not well-done. (c) proposed text guided
image fusion approach: achieving interactive and high-quality fu-
sion image without tedious replacement of models.
Limited by the conditions of environments, the origi-
nally acquired infrared and visible images may suffer from
degradations and show low fusion image quality. The vis-
ible images are susceptible to degradation issues, e.g., low
light, over exposure, etc. The infrared images are inevitably
affected by noise (including thermal, electronic, and envi-
ronmental noise), diminished contrast, and other associated
effects. Current fusion methods lack the capability to adap-
tively solve the degradations, leading to the low-quality fu-
sion image. Furthermore, relying on manual pre-processing
to enhance the image has the problems of flexibility and ef-
ficiency [29]. Therefore, it is of practical interest to study
a model that harmonises degradation-aware processing and
interactive fusion.
Designing a model for individualized degradation to
achieve image enhancement and fusion is feasible. How-
ever, most of image fusion tasks need to be carried out in
various complex conditions around the clock. As shown in
This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.
Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore.
27026
资源评论
