Table 1:Information of data sets Table 2:Data set partition for building,training and testing. DATA SET #PAPERS #CITATIONS #UNIQUE TIMES DATA SETS BUILDING TRAINING TESTING ARXIV-TH 14226 100025 10500 ARXIV-TH 62239 1465 36328 ARXIV-PH 16526 125311 1591 ARXIV-PH 82343 1739 41229 3.3 Convergence Analysis OEM-appr:The approximative OEM with both online In each iteration,the learning algorithm ensures that the ob- 3 and topic steps,where both the topic features and pa- jective function value always decreases.Furthermore,the ob- rameters are adaptively learned over time using the ap- jective function is bounded below by 0.Hence,the learning proximative objective function in(5) algorithm will converge. 4.3 Evaluation Metrics 4 Experiment As in IVu et al..2011bl,we evaluate the above models with the following three metrics: We apply DEM and OEM to two citation networks and com- Average held-out log-likelihood:By taking a logarithm pare the results between them.Furthermore,we will also an- of the likelihood L(B)in (1)for each testing citation alyze the evolution of papers'topic proportions. event,we can get the held-out log-likelihood.We divide the summation of log-likelihood over all testing events in 4.1 Data Sets a batch by the number of events in that batch to get the Since this paper mainly focuses on citation network analysis average held-out log-likelihood of that batch.A higher which is one of the most important applications of dynam- average held-out log-likelihood indicates a better testing ic network analysis,we use two citation data sets,arXiv-TH accuracy. and arXiv-PH,which are crawled from arXiv3.The general Recall of top-K recommendation list:The recall is de- information of these data sets is summarized in Table 1. fined as the fraction of true citation events in the top The arXiv-TH data set is a collection of articles on high K(measured using likelihood)possible citation events. energy physics theory.It spans from 1993 to 1997 with Here K is the cut-point. high-resolution timestamps (millisecond resolution).The arXiv-PH data set is a collection of articles on high energy 。Average held--out normalized rank.:The“rank”for each physics phenomenology.It spans from 1993 to 1997 on a dai- citation event is the position of the true citation in the ly scale.Since the resolution is high enough,we assume that sorted list(in decreasing order of likelihoods)of possi- every new paper joins the network at a unique time and obvi- ble citations.The rank is then normalized by the number ously there can be more than one citation event happening at of possible citations.A lower rank indicates a better pre- each unique time.As mentioned in the previous section,we dictive power. update the topic proportions and parameters batch by batch More specifically,we partition the data sets into mini-batches 4.4 Results and Analysis and each mini-batch contains citation events happening in a As in DEM [Vu et al.,2011b],we split each data set into three span of unique times.For arXiv-TH the number of unique parts which are used for the building phase,training phase times per batch is 100 and for arXiv-PH the number is 20. and testing phase,respectively.The building phase is aimed And the number of events for each mini-batch is about 1500. to build up the statistics of the citation networks and it is rel- atively long in order to mitigate the truncation effect(citation 4.2 Baseline events happening before 1993 is not contained in the data set- We choose the following four models for comparison in the s)and avoid biases.In the training phase,we train the initial model parameters and the topic features.To fully demon- experiments: strate and compare the predictive power of these models,we .DEM:The original DEM with 8 link features and 50 have a relatively long testing phase and the testing phase is topic features.Note that the original DEM is not on- divided into 24 batches.Please remember that the statistics line and the parameters and topic features are fixed after (link features)are dynamically changed in both training and training. testing phases.The sizes(measured by the numbers of cita- tion events)of each phase are listed in Table 2. .OEM-B:The OEM with only online B step,where the To further cut the time cost of OEM,we randomly choose B will be adaptively updated while the topic features a fraction of unique times in every batch to optimize the topic (topic proportions)of each paper will not change over proportions.For example,when optimizing the topic pro- time. portions for paper i after the first batch comes,we randomly .OEM-full:The full OEM with both online B and topic choose 10%(here 10%is what we call citer percentage in the steps,where both the topic features and parameters are rest of this paper)of citers instead of the whole set of them. adaptively learned over time using the objective function This can be used to speed up the computation.In OEM,we in(2). set the hyperparameters =0.1 and citer percentage =10 unless otherwise stated.The effect of these hyperparameters http://snap.stanford.edu/data (citer percentage and A)will be detailed in later experiments.Table 1: Information of data sets DATA SET #PAPERS #CITATIONS #UNIQUE TIMES ARXIV-TH 14226 100025 10500 ARXIV-PH 16526 125311 1591 3.3 Convergence Analysis In each iteration, the learning algorithm ensures that the ob￾jective function value always decreases. Furthermore, the ob￾jective function is bounded below by 0. Hence, the learning algorithm will converge. 4 Experiment We apply DEM and OEM to two citation networks and com￾pare the results between them. Furthermore, we will also an￾alyze the evolution of papers’ topic proportions. 4.1 Data Sets Since this paper mainly focuses on citation network analysis which is one of the most important applications of dynam￾ic network analysis, we use two citation data sets, arXiv-TH and arXiv-PH, which are crawled from arXiv3 . The general information of these data sets is summarized in Table 1. The arXiv-TH data set is a collection of articles on high energy physics theory. It spans from 1993 to 1997 with high-resolution timestamps (millisecond resolution). The arXiv-PH data set is a collection of articles on high energy physics phenomenology. It spans from 1993 to 1997 on a dai￾ly scale. Since the resolution is high enough, we assume that every new paper joins the network at a unique time and obvi￾ously there can be more than one citation event happening at each unique time. As mentioned in the previous section, we update the topic proportions and parameters batch by batch. More specifically, we partition the data sets into mini-batches and each mini-batch contains citation events happening in a span of unique times. For arXiv-TH the number of unique times per batch is 100 and for arXiv-PH the number is 20. And the number of events for each mini-batch is about 1500. 4.2 Baseline We choose the following four models for comparison in the experiments: • DEM: The original DEM with 8 link features and 50 topic features. Note that the original DEM is not on￾line and the parameters and topic features are fixed after training. • OEM-β: The OEM with only online β step, where the β will be adaptively updated while the topic features (topic proportions) of each paper will not change over time. • OEM-full: The full OEM with both online β and topic steps, where both the topic features and parameters are adaptively learned over time using the objective function in (2). 3 http://snap.stanford.edu/data Table 2: Data set partition for building, training and testing. DATA SETS BUILDING TRAINING TESTING ARXIV-TH 62239 1465 36328 ARXIV-PH 82343 1739 41229 • OEM-appr: The approximative OEM with both online β and topic steps, where both the topic features and pa￾rameters are adaptively learned over time using the ap￾proximative objective function in (5). 4.3 Evaluation Metrics As in [Vu et al., 2011b], we evaluate the above models with the following three metrics: • Average held-out log-likelihood: By taking a logarithm of the likelihood L(β) in (1) for each testing citation event, we can get the held-out log-likelihood. We divide the summation of log-likelihood over all testing events in a batch by the number of events in that batch to get the average held-out log-likelihood of that batch. A higher average held-out log-likelihood indicates a better testing accuracy. • Recall of top-K recommendation list: The recall is de- fined as the fraction of true citation events in the top K (measured using likelihood) possible citation events. Here K is the cut-point. • Average held-out normalized rank: The “rank” for each citation event is the position of the true citation in the sorted list (in decreasing order of likelihoods) of possi￾ble citations. The rank is then normalized by the number of possible citations. A lower rank indicates a better pre￾dictive power. 4.4 Results and Analysis As in DEM [Vu et al., 2011b], we split each data set into three parts which are used for the building phase, training phase and testing phase, respectively. The building phase is aimed to build up the statistics of the citation networks and it is rel￾atively long in order to mitigate the truncation effect (citation events happening before 1993 is not contained in the data set￾s) and avoid biases. In the training phase, we train the initial model parameters and the topic features. To fully demon￾strate and compare the predictive power of these models, we have a relatively long testing phase and the testing phase is divided into 24 batches. Please remember that the statistics (link features) are dynamically changed in both training and testing phases. The sizes (measured by the numbers of cita￾tion events) of each phase are listed in Table 2. To further cut the time cost of OEM, we randomly choose a fraction of unique times in every batch to optimize the topic proportions. For example, when optimizing the topic pro￾portions for paper i after the first batch comes, we randomly choose 10% (here 10% is what we call citer percentage in the rest of this paper) of citers instead of the whole set of them. This can be used to speed up the computation. In OEM, we set the hyperparameters λ = 0.1 and citer percentage = 10 unless otherwise stated. The effect of these hyperparameters (citer percentage and λ) will be detailed in later experiments