FRT is a form of biometric1“Biometrics” refers to the measurement of physical characteristics or personal behaviour. Under the understanding that these are unique, and therefore could be used to identify a person, biometrics refers in computer science to the field of authenticating the identity by automatically carrying out these measurements and calculations. Some examples of body parts used for such purposes are face, fingerprints, DNA or the iris. artificial intelligence2For the purposes of this project, we refer to the definition of an “artificial intelligence system” under Article 3 (1) of the European Union Artificial Intelligence Act: “‘AI system’ means a machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments”, together with Recital 12 of the Act, https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L_202401689.
that police use to attempt to identify or characterize a person on the basis of their facial structure.3For more information on the history and development of facial recognition, see Raviv, S, The Secret History of Facial Recognition, Wired, 21 January 2020, https://www.wired.com/story/secret-history-facial-recognition/. Our principles refer to systems using FRT as organized and structured sets of interrelated software, hardware, procedures, datasets and human resources that are deployed – that is, set up and implemented – as a unit. In other words, while the software’s main component involves detecting a facial image and attempting to identify it, it works within a structure, and in conjunction with other hardware and software operated by humans who interpret the outputs produced.
What is FRT used for?
The following are some of the main uses of facial recognition:4See Buolamwini, J et al., Facial Recognition Technologies: A Primer, May 2020, https://people.cs.umass.edu/~elm/papers/FRTprimer.pdf.
Verification
The goal of FRT-based verification is to confirm the identity of a person through the comparison of a single image captured at that moment against a single stored image. For that reason, FRT verification can also be referred to as “one-to-one matching” or “one-to-one comparison”. Examples of this include a person attempting to unlock their smartphone with their face, or when a person’s photograph taken at an airport checkpoint is compared with their passport photograph.
Identification
The goal of FRT-based identification is to determine the identity of an unknown individual whose face is captured on a picture or video by comparing the image of the unknown person against a reference database of images of known people. It can also be referred to as “one-to-many identification”. This is the traditional use of FRT-based surveillance systems, upon which these principles are focused. Examples include police attempting to identify a person by comparing the image of a person taken from a closed-circuit television (CCTV) still or social media against driver’s licence, voter registration or passport holder databases, or comparing the image of a person caught on CCTV against a watchlist of known individuals in a real-time, or live, manner.
Facial attribute classification, estimation and detection
The goal of attribute classification is to obtain information about features that are then used to attempt to measure (such as estimating age) or categorize (such as identifying gender). It can also involve checking whether certain features are present or not (such as detecting if someone is wearing glasses). An example of this use is the categorization of large numbers of people into smaller clusters for marketing purposes,5Kuligowski, K, Facial Recognition Advertising: The New Way to Target Ads to Customers, Business News Daily, 20 October 2023, https://www.businessnewsdaily.com/15213-walgreens-facial-recognition.html. a practice which raises serious concerns about profiling, privacy and data protection.
Facial emotion recognition
The goal of facial emotion recognition is to attempt to infer a person’s feelings or emotions based on their facial expressions. For example, the US retail giant Walmart previously used facial recognition to attempt to identify unsatisfied customers.6Peterson, H, Walmart is developing a robot that identifies unhappy shoppers, Business Insider, 19 July 2017, https://www.businessinsider.com/walmart-is-developing-a-robot-that-identifies-unhappy-shoppers-2017-7. The accuracy and scientific grounding7Stanley, J, Experts Say “Emotion Recognition” Lacks Scientific Foundation, American Civil Liberties Union, 18 July 2019, https://www.aclu.org/news/privacy-technology/experts-say-emotion-recognition-lacks-scientific and Romero, A, AI Emotion Recognition is a Pseudoscientific Multi-billion Dollar Industry, The Algorithmic Bridge, 12 July 2022, https://www.thealgorithmicbridge.com/p/ai-emotion-recognition-is-a-pseudoscientific. of this hugely controversial “pseudoscientific”8Hern, A, “Information commissioner warns firms over ‘emotional analysis’ technologies”, The Guardian, 25 October 2022, https://www.theguardian.com/technology/2022/oct/25/information-commissioner-warns-firms-over-emotional-analysis-technologies. use of facial recognition is highly contested.9Barrett, LF, Adolphs, R, Marsella, S, Martinez, AM & Pollak, SD (2019), “Emotional Expressions Reconsidered: Challenges to Inferring Emotion From Human Facial Movements”, Psychological Science in the Public Interest, 20(1), 1–68. https://doi.org/10.1177/1529100619832930.
Technical notes to accompany these principles
Generally speaking, this use of FRT involves the following steps. However, it should be noted that this set of steps is a simplified version of a very complex process and sequence of actions involving various factors:10Sáez Trigueros, D, Meng, L & Hartnett, M, Face Recognition: From Traditional to DeepLearning Methods, 2018, https://arxiv.org/pdf/1811.00116.
- Face detection: Localization of a face or faces in an image or video and, if any, return of the coordinates of the boxes bounding each of them.
- Face alignment: Modification of the face input (such as scaling or cropping), based on its geometric features, to adapt it to a canonical form in order to allow it to be compared against a database or watchlist of facial images.
- Face representation: Transformation of the pixels in the image into inputs that are useful for computer comparison. This may be a set of templates or, depending on the technique, features or shapes.
- Face matching: Comparison of the input obtained in the previous step against the reference database to assess, with a probability score, the verification, identification or categorization of the face.
