Please write 12 pages excluding the title and references pages. Please included intext citations and references. Please find the attached document named(Need to rewrite

2021 IEEE International Conference on Smart Computing (SMARTCOMP)

F 3S: Free Flow Fever Screening

Kunal Rao, Giuseppe Coviello, Min Feng*, Biplob Debnath, Wang-Pin Hsiung, Murugan Sankaradas, Yi Yang*, Oliver Po, Utsav Drolia* and Srimat Chakradhar

NEC Laboratories America, Inc. Princeton, NJ and San Jose, CA

Abstract—Identification of people with elevated body tem- perature can reduce or dramatically slow down the spread of infectious diseases like COVID-19. We present a novel fever- screening system, F 3S, that uses edge machine learning tech- niques to accurately measure core body temperatures of multiple individuals in a free-flow setting. F 3S performs real-time sensor fusion of visual camera with thermal camera data streams to detect elevated body temperature, and it has several unique features: (a) visual and thermal streams represent very different modalities, and we dynamically associate semantically-equivalent regions across visual and thermal frames by using a new, dynamic alignment technique that analyzes content and context in real-time, (b) we track people through occlusions, identify the eye (inner canthus), forehead, face and head regions where possible, and provide an accurate temperature reading by using a prioritized refinement algorithm, and (c) we robustly detect elevated body temperature even in the presence of personal protective equipment like masks, or sunglasses or hats, all of which can be affected by hot weather and lead to spurious temperature readings. F 3S has been deployed at over a dozen large commercial establishments, providing contact-less, free- flow, real-time fever screening for thousands of employees and customers in indoors and outdoor settings.

Index Terms—Real-time fever screening, thermal and visual imaging, stream fusion, stream compensation, edge computing, deep learning, smart health and well-being applications

I. INTRODUCTION

One of the common symptoms for a person infected with COVID-19 is fever [1]. Reliable and accurate detection of fever helps in isolating potentially infected people. In this paper, we present F3S which screens for people with fever as they move in a free flow setting where individuals need not pause or stop at a kiosk for fever screening. Fig. 1 on the left shows a setup where people need to pause or stop, get their temperature measured and then proceed, while on the right, it shows the free flow movement of people, where temperature is measured as people walk through. By allowing free flow movement, F3S increases the throughput (number of people screened per minute) and avoids over-crowding (which can increase virus transmission).

To measure temperature of individuals, F3S uses thermal and visual cameras, which are placed far away from people, so that temperature measurements can be done from a distance

* Work done while being affiliated with NEC Laboratories America, Inc. 2693-8340/21/$31.00 ©2021 IEEE DOI 10.1109/SMARTCOMP52413.2021.00060

Figure 1: Pause-and-Go (left) vs. Free-Flow (right)

in a contact-less manner. This does not require any human intervention, which is a key concern for COVID-19, since it can spread through close contact of individuals [1]. To do simultaneous fever screening of multiple persons in a free- flow setting, and in real-time, F3S uses edge machine learning techniques to perform sensor fusion of thermal and visual data streams within a few hundred milliseconds. Such low-latency response for multiple persons is not possible when visual and thermal data streams are sent to the cloud for analytics processing. F3S uses resources near the visual and thermal sensors for edge machine learning.

We make the following key contributions:

1) We present F3S, a novel free flow fever screening solution which runs at the edge and through use of deep learning techniques, it enables real-time, simultaneous high-throughput measurement of core body temperature of multiple individuals from a distance, all without any human intervention.

2) We present novel sensor fusion techniques to fuse ther- mal and visual frames to enable accurate temperature measurement of multiple individuals simultaneously.

3) We present a neural network based distance compen- sation model, which enables correct temperature mea- surement at various distances from the thermal camera. Such compensation is necessary because the temperature reported for a person depends on the distance of the person from the thermal camera.

4) We present novel techniques to track temperature of individuals across frames, and at different regions (eye, forehead, face and head), depending on visibility and

ar X

iv :2

10 9.

01 73

3v 1

[ cs

.C V

] 3

S ep

2 02

1

pose of the person; we prioritize, collate and filter alerts for same individual to avoid false positives.

5) We present novel techniques to measure temperature of individuals even when their face is partially covered (for example, if they are wearing masks, sunglasses or hats)

6) Finally, we present a methodology to determine ground truth using thermal and visual sensor data, and to verify accuracy and correctness of temperature measurement.

II. BACKGROUND AND CHALLENGES

Fever screening measures the core body temperature of indi- viduals as they walk into the facility and triggers an alert when the temperature is above a certain pre-configured threshold (Centers for Disease Control and Prevention considers a person to have fever when measured temperature is at least 100.4◦F or 38◦C [1]).

The temperature across a human body is not consistent – head is the hottest, feet are the coldest, and there are variations in between. There are two regions of the human body that can provide most reliable core temperature measurements: 1) inner canthus of the eye – corner of the eye where the upper and lower eyelids meet, and 2) ear canals. For non-invasive febrile temperature screening using thermographic devices, ISO recommends obtaining temperature reading between the eyes of a person [2].

This is the main goal associated with measuring core body temperatures remotely, i.e. non-invasively and without human proximity, using thermographic sensors – detecting this region reliably, and measuring accurate core body temperature.

There are two main challenges that F3S addresses in achieving this goal.

1) Free Flow: How to reliably measure core body temper- ature while people are moving, possibly occluding each other, and are not asked to stop and look at the sensor?

2) Eye-region Occlusion: How to adapt to eye-region obstructions, such as glasses, caps, masks; measure core body temperature and still allow uninterrupted flow of entry?

III. RELATED WORK

Infrared thermography has been deployed at the quarantine stations for detecting elevated body temperature in the last two decades [3]–[5]. There are many research works that combine RGB and thermal images to extract multiple vital signs (i.e. body temperature, heart rate, respiration rate, blood volume pulse, etc.) to identify people with febrile conditions [3], [6]–[8]. However, these works have been designed for the pause-and-go like scenario, where an individual pauses for a few seconds, stands still and looks straight at the camera. Even commercial solutions like [9]–[14] work well only in pause-and-go scenarios and not in a free-flow setting. They assume that the face and eye-region is clearly visible with no obstruction, which may not be the case in a free-flow scenario, where people are screened as they walk through and face and/or eye-region may or may not be always visible due to occlusion.

Haghmohammadi et al. propose a method for measuring body temperature for an indoor moving crowd [15]. However, it also does not work when the face is not clearly visible. In addition, it has not been tested thoroughly for crowd scenario. Somboonkaew et al. propose a mobile-platform for massive human temperature screening in large public areas based on the infrared forehead temperature using an IR camera and a mobile phone [16]. They align the RGB image to the thermal image to find the target area of the temperature measurement. For the image alignment, they use image cropping and image scaling based on field of view and image resolution of the RGB and thermal cameras. This alignment procedure, however, is very simple and does not work well when faces are occluded. In contrast, F3S uses sophisticated method to dynamically align visual and thermal image pairs (explained in Section V-B2), which enables accurate temperature measurement in a free-flow setting, even when there is occlusion.

IV. F3S OVERVIEW

F3S is a fever screening system designed to operate in a free-flow manner i.e. individuals are not required to pause/stop in order for the temperature to be measured. Instead, their temperatures is measured automatically from a distance as they cross the field of view of the camera. A key aspect of our system is that it does not require any human intervention, which is a key concern for COVID-19, since it can spread through close contact of individuals [1]. Our solution works in real-time and is able to handle a high volume and rate of flow.

Fig. 2 shows the setup for deployment of F3S. Arrow shows the direction of movement of people from the entrance into the building. Cameras are located further away pointing towards the entrance so that a large enough field-of-view is captured. As people walk into the building, their temperatures are measured and displayed on the screen for the operator to monitor.

If the temperature of a person is above the configured threshold, then an alert is triggered and the operator can

Figure 2: F3S Deployment

request the individual to step aside and proceed for secondary screening. F3S acts as an initial screening solution and final screening is done using a medical-grade thermometer. Physical distance between operator and target individuals is constantly maintained, as the operator does not need to come in close contact with the people. In fact, the operator need not be present at the location physically as well, everything can be monitored from a central location in the control room and if an alert is triggered, the individual can be notified over an audio speaker to step aside, thereby avoiding the need for any human to physically intervene.

V. F3S DESIGN

The primary goal of any fever screening system is to measure a person’s temperature in their eye region, particularly inner canthus area, to produce the most reliable measurement [2]. Using a thermal sensor alone to detect this region and measure the temperature is possible, but the accuracy of detection is poor [17]. In order to overcome this limitation, we combine visual sensor data with thermal sensor data to get accurate readings of the temperature of every person. Visual sensors produce higher dimensional data than thermal sensors (higher resolution and channels in image data). Therefore, visual camera streams allow for more accurate detection of persons, their faces and eye-regions. However, visual cameras lack information about temperature, which is available in the thermal stream data. We use deep learning models to perform sensor fusion of visual and thermal streams to associate temperatures in thermal data with corresponding objects in the visual stream.

Fig. 3 shows a high-level design and workflow of F3S. There are two streams of data coming into our system i.e. thermal data stream and visual data stream. Both these streams relate to the same scene, one containing the visual RGB frame and the other containing the thermal frame of the scene. Goal. The system’s goal is to detect the core body temperature of people as they move in a free-flow manner, from their eye region when possible, or other appropriate body regions.

In order to achieve this goal, we split the design of F3S into three components, each of them handling specific challenges.

1) Person Tracking: The system needs to detect and track people, while also identifying various regions for temperature measurement as they walk through.

2) Frame Fusion: The system needs to correlate the identified regions in the visual frame with regions in

Figure 3: F3S design and workflow

the thermal frame, from where temperatures can be extracted.

3) Fever Screening: The system needs to decide if a person actually has fever or not, based on the multiple detections it possibly makes across frames.

Next, we discuss each of these components in detail.

A. Person Tracking

This component of F3S detects people and tracks them as they move in a free-flow setting. We use a proprietary neural network based model [18], which detects the face, head and body of a person. Depending on whether the person is wearing glasses, mask, hat, or a combination of these, and the pose and angle – none, one or more of the above detections are possible for a person in each frame. After detection, tracking of the person is a key component in our system to achieve high accuracy when people are not pausing or stopping. Having a unique tracking ID for the detected regions of a person allows our system to correlate and refine temperature measurements across frames for them as they walk through.

For tracking purpose, person tracking component maintains a cache, called as person cache, to store details of most re- cently detected individuals. Specifically, for these individuals, the latest bounding box location of body, head and face, along with their snapshots is maintained. A person remains in the cache until there are body, face or head detections available for the person. If there are no detections available for some configurable period of time e.g. 10 seconds, then the person is removed from the cache (since the person may have left and therefore there are no detections). In order to assign unique ID for an individual, each of the available detections i.e. body, face or head is checked one by one.

Our tracking algorithm is described in Algorithm 1. In the first loop, detected body is compared with all previously stored bodies in the cache to see if there is any match. For body match, we consider the bounding box location and image similarity. If the overlap in bounding box is high i.e. above a pre-configured threshold and the image similarity is also high (above a pre-configured threshold), then it is considered as a match. If a body match is found, then the ID of the body in cache is assigned to the detected body and details of the body i.e. bounding box and snapshot is updated in the cache. If there is no match found, then a new ID is assigned to the body and the detected body is added to the cache.

The second loop uses proprietary neural network based face recognition model [18] to compare and match the face with all faces in the cache. If the match score is above a pre-configured threshold, then it is considered as a face match and the ID of the face in the cache is assigned to the detected face, and face details in cache is updated. Since accuracy of face recognition model is higher, we use this to overwrite the ID for body and assign the same ID as face, to the body. If there is no face match found, then ID of the body, if available, is assigned to the face, otherwise a new ID is assigned to the face and the detected face is added to the cache.

Data: Body, Head, Face in frame Result: Body, Head, Face with tracking ID foreach detected body do

if body matches any body in person cache then assign the ID in cache to the body;

else assign a new ID to the body;

end end foreach detected face do

if face matches a face in person cache then assign the ID of the cache to the face; if face in a body then

assign the ID of the face to the body; end

else if face in a body then

assign the ID in body to the face; else

assign a new ID to the face; end

end end foreach detected head do

assign ID of a face or body, or a new ID to the head; end

Algorithm 1: Person tracking algorithm

The third loop assigns an ID to the head detection. We do not use image similarity and bounding box location match for head because the head detection has a low detection rate compared to body and less accuracy for similarity match compared to face match. We assign the same ID as the face or body, if available, otherwise we assign a new ID to the head. If both face and body ID are available, then we give priority to face ID.

Thus by going through this procedure, where we check the available body, face or head detections one by one, we are able to track and assign unique ID to individuals, which enables F3S to correlate and refine temperature measurement for an individual as the person walks through in a free-flow manner.

B. Frame Fusion

As mentioned above, F3S uses both, visual (RGB) and thermal sensors to achieve high accuracy and high throughput fever screening. The system detects persons on the visual frames and then finds the temperature of the person from the thermal frame. To do this, a mapping from the visual frame to the thermal frame is required, i.e. for a given point (xv,yv) in the visual frame, what is the corresponding (xt,yt) in the thermal frame. Now, (xv,yv) == (xt,yt) if the sensors have the exact same viewpoint. However, even though these sensors are assembled in a single unit, they are placed side- by-side, with discernible distance between them. Such a setup introduces a difference in the sensor viewpoints, which implies that (xv,yv)! = (xt,yt). Instead, the points are related through a function – (xv,yv) = falign(xt,yt). The system needs to estimate falign so that a point from the visual frame can be mapped to the correct point in the thermal frame to read the associated temperature.

1) Manual Offset Approach: A straightforward way to get approximate alignment is to perform static scaling and translation. Let IV be visual sensor data, IT be thermal sensor data and IAT be thermal sensor data aligned to visual image. Then alignment function falign can be defined as follows (where tx is horizontal offset, ty is vertical offset, Sx is horizontal scale factor and Sy is vertical scale factor).

falign =

 1 00 1 tx ty

  * [Sx 0

0 Sy

] With above transformation function, it is possible to get

temperature measurement in known area of visual ROI using data from IAT = falign(IT ).

However, this approach produces correct alignment only for a shallow depth plane, and approximate alignment for neighbouring pixels in that area of that plane. It does not align the entire frame. Pixels representing depth planes further or much closer from the correctly aligned image plane have larger misalignment. Alignment errors can be about 100+ pixels (in both x and y direction) when the person is closer or further to camera. This leads to inaccurate temperatures being assigned to visual regions.

2) Dynamic Frame Alignment: Relative distortion between sensor images vary depending upon the depth plane. As the individual walks towards the camera, multiple temperature readings have to be taken to take into account head and face pose, occlusion with other individuals and objects. Since typical width of free flow traffic is about 6 to 8 feet wide and with horizontal field of view angle spans about 90 degrees or more (due to measurement zone constraints), it leads to non- uniform misalignment in horizontal plane as the individuals are found off camera’s optical axis. Alignment issues is shown in Fig. 4 and the alignment error at various distances along horizontal and vertical plane is shown in Fig. 5. Corresponding measurement inaccuracy and required correction is depicted in Fig. 6. F3S dynamically aligns visual and thermal image pairs

for all people in field of view using the flow shown in Fig. 7. Since thermal image has limited features to extract and limited correlation to visual image, features in boundary of person’s head are matched in visual and thermal spectrum. To do this, first background is estimated in the images and subtracted and foreground mask is obtained. ORB features [19] of foreground mask is obtained and matched using a brute

Figure 4: Non-uniform misalignment

Figure 5: Alignment error

Figure 6: Measurement inaccuracy due to Misalignment

force Hamming matcher to identify matching feature points. Using the matching feature points with high confidence score across both pairs, Homography matrix is obtained for each pair. Object pairs are rectified with respect to each other using Homography matrix in Spatial Fusion module.

A by-product of this alignment is distance measurement.

Figure 7: Dynamic Frame Alignment

The thermal and visual cameras are placed in a stereo ar- rangement with a small baseline. This allows the application of epipolar geometry and stereo vision principles to calcu- late depth-from-disparity [20]. Typically, the critical issue in assessing depth from cross-spectral stereo is finding corre- sponding points across the two spectra. We already solve this problem during the alignment process and using that, we obtain the depth and use it in subsequent module for temperature correction.

C. Fever Screening

Once visual and thermal frame fusion is completed, Fever- screening module then processes the fused frame to detect people with fever. Four key challenges that this module solves are:

1) Detecting and measuring temperature within the sensor’s recommended measurement zone

2) Prioritizing temperature measurement region depending on available detections for an individual within a frame

3) Temperature correction based on distance from camera 4) Prioritized refinement of measured temperatures across

frames, and alerting for those with fever 1) Detection and Measurement in Capture Zone.: Mea-

sured temperature from the thermal sensor varies depending on the distance from the camera. This is because of the atmospheric composition between the person and the camera. Gases and particles present in the atmosphere absorb some of the emitted infrared radiation from the person. Farther the distance, more is the absorption and therefore less is the measured temperature. This variation of temperature with distance is shown in Fig. 8 for an individual along the optical axis and 2 feet off from the optical axis. Due to these atmospheric factors, thermal imaging sensors have a recommended measurement zone in terms of distance from the sensor. To allow measurement within this recommended zone, a capture zone is set up i.e. a zone in which temperature measurements for individuals starts as they enter and stops when they leave. Along with capture zone, a Region of Interest (ROI) within the camera view is also configured. This can be used to tightly control the region within which temperature is measured for individuals, thus keeping it clutter-free for the operator. Note that within the ROI, people may be too far or too near i.e. may be within or outside the capture zone.

Figure 8: Temperature variation with distance

Data: Fused stream Result: Temperatures for detected people discardIfFrameOutOfOrder(); discardPersonsOutsideROI(); /* process detected persons */ foreach detected person within ROI do

if person within capture zone then if person present in cache then

measureTemperatureAndUpdateInCache(); else

if person entered capture zone then measureTemperatureAndAddInCache();

end end

end renderFrameWithAnnotations(); /* calculate/revise temperature and send

alert */ sendTemperatureAlert(); removeExpiredFromCache();

Algorithm 2: Fever Screening algorithm

Algorithm 2 shows the procedure followed in determining the temperature of individuals as they walk through in the capture zone. The first step is to discard out-of-order frames. Next, all detected persons who lie outside the configured ROI are removed. After removal of any person outside the ROI, the remaining individuals are processed one by one as they enter and leave the capture zone.

Fever-screening module maintains a cache of recently seen individuals. Tracking ID of the individual is used as an identifier to determine if the person is new or previously seen. For a previously seen person, the new temperature reading is measured for the current frame and updated in cache corresponding to the tracking ID of the individual. For a new person, check is performed to see if the person has entered the capture zone. If it is determined that the person has entered the capture zone, then temperature for the person is measured and a new person with new tracking ID is added in cache. Note that the temperature for a person is added or updated in cache only if it is within the acceptable human temperature range, so as to avoid any spurious temperature readings. Also, note that we start temperature measurement for an individual when he/she enters the capture zone and continue to monitor and measure until the individual leaves the capture zone.

After all individuals are processed, the frame is annotated with the temperatures for individuals and rendered, so that the operator can see the live view of the feed with temperatures of individuals annotated on the live stream. After frame rendering is done, all individuals in cache are processed one by one to (a) determine temperature for the individual across multiple readings and send alert if temperature is above a certain configured threshold and (b) remove any expired entries from cache i.e. remove any individuals in cache after a configurable period of time, after they have left.

2) Prioritized temperature measurement.: Algorithm 3 shows the procedure followed to prioritize and measure the temperature for a person within a frame. Highest priority is given to temperature measurement at eye and forehead region,

Data: Person detection within fused frame Result: Core body temperature for the person temperature ←− 0.0; measured ←− false; if person has face detection then

if face has eye detection then temperature ←− getTemperatureAtEyeForeheadRegion();

measured ←− true; else

temperature ←− getTemperatureAtFaceRegion();

measured ←− true; end

end if NOT measured AND person has head detection then

temperature ←− getTemperatureAtHeadRegion(); measured ←− true;

end if measured then

temperature ←− getCorrectedTemperature(); end return temperature Algorithm 3: Prioritized temperature measurement

followed by face and finally to head region. Within each of these regions, a configurable area is chosen and temperatures of pixels within that area is obtained from thermal sensor data. Among all the temperature values in the area, maximum among those is chosen as the raw measured temperature. We decided to choose the maximum value so as to avoid a false negative i.e. missing an individual who might have fever.

3) Temperature correction: As mentioned in section V-C1, the measured temperature of an individual varies depending on the distance from the camera and this variation is non- linear. To correct for this variation in measured temperature with distance, F3S employs a neural network based distance compensation model, which can be used when a black body is present as part of the deployment. Black body provides a known reference temperature to ensure accurate temperature reading. This problem is framed as a regression problem and we use Multi-layer Perceptron (MLP) to solve it. The model is a feed-forward neural network, to which the input parameters are the distance and the corresponding measured temperature at that distance. The output from the model is the predicted true temperature of the individual. The model architecture is shown in Fig. 9, with input layer, hidden layer and an output layer.

At the time of model training, as people walk through, the

Figure 9: Distance compensation model

measured temperature at the black body is considered as the ground truth i.e. the true temperature of the person. Using this as the ground truth, the loss function is set as the mean squared error given by equation 1, where Yi is the true temperature and Ŷi is the predicted temperature.

L = 1

n

i=n∑ i=1

( Yi − Ŷi

)2 (1)

We use Adam [21] optimizer with following hyper- parameter settings: α of 0.001, β1 of 0.9, β2 of 0.999 and � of 1e−07 for training. Rectified Linear Unit (ReLU) is used as activation function and the model is trained until 100 epochs with batch size of 1. The split between training and test data is 70:30. After the model is trained, the evaluation resulted in a mean squared error of 0.018 on test data, indicating a very high accuracy for the trained distance compensation model.

4) Prioritized refinement and alerting: As an individual walks through, multiple temperature readings of the person are recorded across frames. Based on these readings algorithm 4 shows the procedure followed to prioritize and refine temper- ature measurement and send alert. To prioritize temperature measurement, highest priority is given to eyes and forehead, followed by face and head. Maximum temperature for the available highest priority region is recorded as the temperature for the person (to avoid missing a person with fever). To refine the measured temperature, as person is seen across frames, if temperature readings of higher priority region is measured, then the previous recorded temperature from lower priority region, is refined and updated with the new one. With respect to alerting, F3S uses tracking ID to avoid sending repeated alerts for same individual. An alert is sent the first time measured temperature for a person is greater than the configured threshold, and after that an alert is sent only if higher priority region temperature is measured or if the delta increase in the temperature is above a configured threshold.

VI. AUTO-CALIBRATION

As the environmental conditions change, the thermal read- ings from the camera start to drift from actual tempera- ture, thereby producing incorrect temperature readings, which ultimately results in incorrect temperature measurement for individuals. Camera vendors provide several parameters to calibrate and correct for this drift and maintain original tem- peratures. This correction however must be done manually, which is not practical to do in a real deployment where environmental conditions may change frequently. In order to do this automatically, F3S employs a separate module to automatically detect the change and quickly adjust the camera parameters to maintain accurate temperature readings produced by the camera.

In order to detect the change automatically, a black body is used as a reference object and set to a known reference temperature in the field of view of the camera. The temper- ature readings coming from the region of the black body is continuously monitored and any change in the temperature

Data: Cache of recently seen individuals Result: Prioritized, refined temperature and alert foreach cache entry do

if minimum number of readings present then max head temperature ←− getMaxHeadTemperatureReading();

max face temperature ←− getMaxFaceTemperatureReading();

max eye forehead temperature ←− getMaxEyeForeheadTemperatureReading();

if higher priority region temperature is available then

updatePersonTemperatureWithHigherPriorityReading();

else updatePersonTemperatureR

Collepals.com Plagiarism Free Papers

Are you looking for custom essay writing service or even dissertation writing services? Just request for our write my paper service, and we'll match you with the best essay writer in your subject! With an exceptional team of professional academic experts in a wide range of subjects, we can guarantee you an unrivaled quality of custom-written papers.

Get ZERO PLAGIARISM, HUMAN WRITTEN ESSAYS

Why Hire Collepals.com writers to do your paper?

Quality- We are experienced and have access to ample research materials.

We write plagiarism Free Content

Confidential- We never share or sell your personal information to third parties.

Support-Chat with us today! We are always waiting to answer all your questions.