Fig.5.(a)Original image.(b)Detection using (R.G.B)color space.(c)detection using chromaticity coordinates (r.g)and the lightness variable,s. Although using chromaticity coordinates helps in the sup- 0 pression of shadows,they have the disadvantage of losing Fig.2.Effect of the second stage of detection on suppressing false lightness information.Lightness is related to the differences detections.(a)Original image.(b)First stage detection result.(c) in whiteness,blackness,and grayness between different ob- Suppressing pixels with high displacement probabilities.(d)Result jects [23].For example,consider the case where the target using component displacement probability constraint. wears a white shirt and walks against a gray background.In this case,there is no color information.Since both white and gray have the same chromaticity coordinates,the target may not be detected. To address this problem,we also need to use a measure of lightness at each pixel.We use s =R+G+B as a lightness measure.Consider the case where the background is completely static,and let the expected value for a pixel Fig.3.(a)Original image.(b)Result after the first stage of be (r,g,s).Assume that this pixel is covered by shadow in detection.(c)Result after the second stage. frame tand let(r,,st)be the observed value for this pixel at this frame.Then,it is expected that a<(st/s)<1.That is,it is expected that the observed value st will be darker than the normal value s up to a certain limit,as<st,which corresponds to the intuition that at most a fraction (1-a) of the light coming to this pixel can be reduced by a target shadow.A similar effect is expected for highlighted back- ground,where the observed value can be brighter than the Fig.4.(a)Original image.(b)Detection using (R,G.B)color expected value up to a certain limit.Similar reasoning was space.(c)Detection using chromaticity coordinates(r.g). used by [24]. In our case,where the background is not static,there is 3)Working With Color:The detection of shadows as part no single expected value for each pixel.Let Abe the sample values representing the background for a certain pixel,each of the foreground regions is a source of confusion for subse- quent phases of analysis.It is desirable to discriminate be- represented as i=(ri,gi,si),and let t=(rt,g,st)be the observed value at frame t.Then,we can select a subset tween targets and their shadows.Color information is useful for suppressing shadows from the detection by separating B C A of sample values that are relevant to the observed lightness st.By relevant,we mean those values from the color information from lightness information.Given three sample which,if affected by shadows,can produce the ob- color variables,R,G,and B,the chromaticity coordinates served lightness of the pixel.That is. are rR/(R+G+B),gG/(R+G+B),and b=B/(R+G+B),wherer+g+b=1 [22].Using chro- maticity coordinates for detection has the advantage of being B={ AAa≤年≤B more insensitive to small changes in illumination that arise due to shadows.Fig.4 shows the results of detection using Using this relevant sample subset,we carry out our kernel both(R,G,B)space and (r,g)space.The figure shows that calculation,as described in Section III-B,based on the two- using the chromaticity coordinates allows detection of the dimensional(2-D)(r,g)color space.The parameters oand target without detecting its shadow.It must be noticed that are fixed over all the image.Fig.5 shows the detection results the background subtraction technique we describe in Section for an indoor scene using both the(R,G,B)color space and III-B can be used with any color space(e.g.,HSV,YUV,etc.). the (r,g)color space after using the lightness variable s to 1156 PROCEEDINGS OF THE IEEE,VOL.90,NO.7,JULY 2002Fig. 2. Effect of the second stage of detection on suppressing false detections. (a) Original image. (b) First stage detection result. (c) Suppressing pixels with high displacement probabilities. (d) Result using component displacement probability constraint. Fig. 3. (a) Original image. (b) Result after the first stage of detection. (c) Result after the second stage. Fig. 4. (a) Original image. (b) Detection using (R; G; B) color space. (c) Detection using chromaticity coordinates (r; g). 3) Working With Color: The detection of shadows as part of the foreground regions is a source of confusion for subse￾quent phases of analysis. It is desirable to discriminate be￾tween targets and their shadows. Color information is useful for suppressing shadows from the detection by separating color information from lightness information. Given three color variables, and , the chromaticity coordinates are and , where [22]. Using chro￾maticity coordinates for detection has the advantage of being more insensitive to small changes in illumination that arise due to shadows. Fig. 4 shows the results of detection using both space and space. The figure shows that using the chromaticity coordinates allows detection of the target without detecting its shadow. It must be noticed that the background subtraction technique we describe in Section III-B can be used with any color space (e.g., HSV, YUV, etc.). Fig. 5. (a) Original image. (b) Detection using (R; G; B) color space. (c) detection using chromaticity coordinates (r; g) and the lightness variable, s. Although using chromaticity coordinates helps in the sup￾pression of shadows, they have the disadvantage of losing lightness information. Lightness is related to the differences in whiteness, blackness, and grayness between different ob￾jects [23]. For example, consider the case where the target wears a white shirt and walks against a gray background. In this case, there is no color information. Since both white and gray have the same chromaticity coordinates, the target may not be detected. To address this problem, we also need to use a measure of lightness at each pixel. We use as a lightness measure. Consider the case where the background is completely static, and let the expected value for a pixel be . Assume that this pixel is covered by shadow in frame and let be the observed value for this pixel at this frame. Then, it is expected that . That is, it is expected that the observed value will be darker than the normal value up to a certain limit, , which corresponds to the intuition that at most a fraction of the light coming to this pixel can be reduced by a target shadow. A similar effect is expected for highlighted back￾ground, where the observed value can be brighter than the expected value up to a certain limit. Similar reasoning was used by [24]. In our case, where the background is not static, there is no single expected value for each pixel. Let be the sample values representing the background for a certain pixel, each represented as , and let be the observed value at frame . Then, we can select a subset of sample values that are relevant to the observed lightness . By relevant, we mean those values from the sample which, if affected by shadows, can produce the ob￾served lightness of the pixel. That is, Using this relevant sample subset, we carry out our kernel calculation, as described in Section III-B, based on the two￾dimensional (2-D) color space. The parameters and are fixed over all the image. Fig. 5 shows the detection results for an indoor scene using both the color space and the color space after using the lightness variable to 1156 PROCEEDINGS OF THE IEEE, VOL. 90, NO. 7, JULY 2002