EEE TRANSACTIONS ON IMAGE PROCESSING.VOL.XX,NO.X.XXX 2019 2 hashing (SePH)[38],supervised matrix factorization hash- A.Continuous Cross-Modal Hashing ing (SMFH)[39],binary set hashing (BSH)[40].deep cross-modal hashing (DCMH)[34],cross-modal Hamming Continuous CMH methods usually adopt relaxation strategy hashing (CMHH)[41].and generalized semantic preserving for learning.More specifically,these methods adopt relaxation hashing(GSPH)[42]. strategy to learn continuous representation at the first stage, and then utilize some rounding techniques to generate discrete According to the training strategy,existing CMH methods binary codes at the second stage.Representative continuous can be divided into two categories:relaxation-based contin- CMH methods include CMFH [27],BSE [40].SCM [26]. uous methods and discrete methods.Hashing is essentially a SMFH [39]and GSPH [421.CMFH is an unsupervised CMH discrete learning problem.To avoid the difficulty caused by discrete learning,relaxation-based continuous methods try to method which adopts collective matrix factorization to learn cross-view hash functions.BSE,SCM,SMFH and GSPH solve a relaxed continuous problem with some relaxation strat- egy.Representative continuous methods include CMFH [27], are supervised CMH methods.BSE tries to preserve the cross view hashing (CVH)[43].SCM [26].SMFH [39]and inter-modal and intra-modal similarity by learning two pro- jections and taking the geometric structures of each modality GSPH [421.Discrete methods try to directly solve the discrete problem without continuous relaxation.Representative dis- into account.SCM learns two hash functions by integrating crete methods include CCA-ITO [61.ACQ [351.MLBE [37]. semantic labels into the learning procedure.To perform effi- predictable dual-view hashing (PDH)[44]and SePH [38]. cient training and fully utilize supervised information,SCM In general,the training of relaxation-based continuous meth- proposes an approximation method to avoid explicit computa- tion of the pairwise similarity matrix.SMFH is the supervised ods is faster than discrete methods,but the accuracy of version of CMFH which integrates supervised information into relaxation-based continuous methods is not satisfactory.On the contrary,the accuracy of discrete methods is typically better learning procedure to further improve retrieval performance. GSPH tries to design a generalized hashing framework to than relaxation-based continuous methods,but the training of handle a wide range of scenarios discrete methods is time-consuming. In this paper,we propose a novel CMH method, One shortcoming of the continuous methods is that the re- laxation procedure might result in a sub-optimal solution [45] called discrete latent factor model based cross-modal hashing (DLFH),for cross modal similarity search.The con- tributions of DLFH are outlined as follows: B.Discrete Cross-Modal Hashing DLFH is a supervised CMH method,and in DLFH a novel discrete latent factor model is proposed to model Discrete CMH methods try to directly learn binary codes the supervised information. without discarding discrete constraints.Representative discrete DLFH is a discrete method which can directly learn the CMH methods include CCA-ITQ [46],ACQ [35],MLBE [37], binary hash codes without continuous relaxation. PDH [44].SePH [38]and DCMH [34].CCA-ITO.ACO and .A novel discrete learning algorithm is proposed for PDH are unsupervised CMH methods.CCA-ITQ and ACQ DLFH,which can be proved to be convergent.Further- utilize the dimension reduction technologies,e.g.,canoni- more,the implementation of DLFH is simple. cal correlation analysis (CCA)and neighborhood preserving The training (learning)of DLFH is still efficient although embedding (NPE)[47],to embed multimodal data into one DLFH is a discrete method. common subspace.Then,they minimize the quantization error Experiments on real datasets show that DLFH can achieve to learn binary codes.PDH adopts max-margin theory to significantly better accuracy than existing methods,in- learn binary codes.By using an iterative optimization method cluding both relaxation-based continuous methods and based on block coordinate descent.PDH tries to maintain existing discrete methods.Experimental results also show the predictability of the binary codes.MLBE,SePH and that the training speed of DLFH is comparable to that of DCMH are supervised CMH methods.MLBE leverages a relaxation-based continuous methods,and is much faster probabilistic latent factor model to learn binary codes which are devised as binary latent factors and designs an alternating than that of existing discrete methods learning algorithm to learn binary latent factors (i.e.,binary The rest of this paper is organized as follows.In Section II, codes).Although MLBE is a supervised CMH method,the we briefly review the related works.In Section III.we de- complexity of MLBE is extremely high.Hence,the training scribe the notations and problem definition.In Section IV,we of MLBE is extremely time-consuming and it cannot scale present our DLFH in detail,including model formulation and to large-scale datasets.SePH aims to transform semantic learning algorithm.In Section V,we carry out experiments for affinities information between training data into a probability evaluation on three widely used datasets.At last,we conclude distribution by minimizing the Kullback-Leibler divergence. the paper in Section VI. Furthermore,SePH utilizes a kernel logistic regression method as an out-of-the-sample strategy to learn binary code for un- II.RELATED WORK seen data.SePH relaxes binary code as real-value continuous codes and imposes a quantization error term to learn binary In this section,we briefly review the related works of codes.In the meantime,the time complexity of SePH is also cross-modal hashing.including continuous cross-modal hash-high.DCMH is a deep learning based cross-modal hashing ing and discrete cross-modal hashing. method which integrates feature learning and binary codeIEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. XX, NO. X, XXX 2019 2 hashing (SePH) [38], supervised matrix factorization hash￾ing (SMFH) [39], binary set hashing (BSH) [40], deep cross-modal hashing (DCMH) [34], cross-modal Hamming hashing (CMHH) [41], and generalized semantic preserving hashing (GSPH) [42] . According to the training strategy, existing CMH methods can be divided into two categories: relaxation-based contin￾uous methods and discrete methods. Hashing is essentially a discrete learning problem. To avoid the difficulty caused by discrete learning, relaxation-based continuous methods try to solve a relaxed continuous problem with some relaxation strat￾egy. Representative continuous methods include CMFH [27], cross view hashing (CVH) [43], SCM [26], SMFH [39] and GSPH [42]. Discrete methods try to directly solve the discrete problem without continuous relaxation. Representative dis￾crete methods include CCA-ITQ [6], ACQ [35], MLBE [37], predictable dual-view hashing (PDH) [44] and SePH [38]. In general, the training of relaxation-based continuous meth￾ods is faster than discrete methods, but the accuracy of relaxation-based continuous methods is not satisfactory. On the contrary, the accuracy of discrete methods is typically better than relaxation-based continuous methods, but the training of discrete methods is time-consuming. In this paper, we propose a novel CMH method, called discrete latent factor model based cross-modal hashing (DLFH), for cross modal similarity search. The con￾tributions of DLFH are outlined as follows: • DLFH is a supervised CMH method, and in DLFH a novel discrete latent factor model is proposed to model the supervised information. • DLFH is a discrete method which can directly learn the binary hash codes without continuous relaxation. • A novel discrete learning algorithm is proposed for DLFH, which can be proved to be convergent. Further￾more, the implementation of DLFH is simple. • The training (learning) of DLFH is still efficient although DLFH is a discrete method. • Experiments on real datasets show that DLFH can achieve significantly better accuracy than existing methods, in￾cluding both relaxation-based continuous methods and existing discrete methods. Experimental results also show that the training speed of DLFH is comparable to that of relaxation-based continuous methods, and is much faster than that of existing discrete methods. The rest of this paper is organized as follows. In Section II, we briefly review the related works. In Section III, we de￾scribe the notations and problem definition. In Section IV, we present our DLFH in detail, including model formulation and learning algorithm. In Section V, we carry out experiments for evaluation on three widely used datasets. At last, we conclude the paper in Section VI. II. RELATED WORK In this section, we briefly review the related works of cross-modal hashing, including continuous cross-modal hash￾ing and discrete cross-modal hashing. A. Continuous Cross-Modal Hashing Continuous CMH methods usually adopt relaxation strategy for learning. More specifically, these methods adopt relaxation strategy to learn continuous representation at the first stage, and then utilize some rounding techniques to generate discrete binary codes at the second stage. Representative continuous CMH methods include CMFH [27], BSE [40], SCM [26], SMFH [39] and GSPH [42]. CMFH is an unsupervised CMH method which adopts collective matrix factorization to learn cross-view hash functions. BSE, SCM, SMFH and GSPH are supervised CMH methods. BSE tries to preserve the inter-modal and intra-modal similarity by learning two pro￾jections and taking the geometric structures of each modality into account. SCM learns two hash functions by integrating semantic labels into the learning procedure. To perform effi- cient training and fully utilize supervised information, SCM proposes an approximation method to avoid explicit computa￾tion of the pairwise similarity matrix. SMFH is the supervised version of CMFH which integrates supervised information into learning procedure to further improve retrieval performance. GSPH tries to design a generalized hashing framework to handle a wide range of scenarios. One shortcoming of the continuous methods is that the re￾laxation procedure might result in a sub-optimal solution [45]. B. Discrete Cross-Modal Hashing Discrete CMH methods try to directly learn binary codes without discarding discrete constraints. Representative discrete CMH methods include CCA-ITQ [46], ACQ [35], MLBE [37], PDH [44], SePH [38] and DCMH [34]. CCA-ITQ, ACQ and PDH are unsupervised CMH methods. CCA-ITQ and ACQ utilize the dimension reduction technologies, e.g., canoni￾cal correlation analysis (CCA) and neighborhood preserving embedding (NPE) [47], to embed multimodal data into one common subspace. Then, they minimize the quantization error to learn binary codes. PDH adopts max-margin theory to learn binary codes. By using an iterative optimization method based on block coordinate descent, PDH tries to maintain the predictability of the binary codes. MLBE, SePH and DCMH are supervised CMH methods. MLBE leverages a probabilistic latent factor model to learn binary codes which are devised as binary latent factors and designs an alternating learning algorithm to learn binary latent factors (i.e., binary codes). Although MLBE is a supervised CMH method, the complexity of MLBE is extremely high. Hence, the training of MLBE is extremely time-consuming and it cannot scale to large-scale datasets. SePH aims to transform semantic affinities information between training data into a probability distribution by minimizing the Kullback-Leibler divergence. Furthermore, SePH utilizes a kernel logistic regression method as an out-of-the-sample strategy to learn binary code for un￾seen data. SePH relaxes binary code as real-value continuous codes and imposes a quantization error term to learn binary codes. In the meantime, the time complexity of SePH is also high. DCMH is a deep learning based cross-modal hashing method which integrates feature learning and binary code