Skip to content

Search

Our search API allows you to search for registered persons by using an image.

Given an image of a person, search answers the following two questions:

  1. Are they a registered person?
  2. Which registered person are they?

This facilitates a wide variety of use-cases ranging from Attendance Tracking, Building Security, etc.

Sending a search request

The following snippet shows the process of creating a search request:

from opencv.fr.search.schemas import SearchRequest, SearchMode
from pathlib import Path

# As before, you can also use OpenCV or Pillow images
image_base_path = Path("sample_images")
image_path = image_base_path / "varun.jpg"

search_request = SearchRequest([image_path], min_score=0.7,
                               collection_id=None,
                               search_mode=SearchMode.FAST)

In addition to supplying up to 3 images in a list, you can optionally specify a collection_id, min_score, and SearchMode.

The collection_id if not None, will restrict the search to only persons who belong to the specified collection.

The min_score is a similarity threshold ranging between 0 and 1. When specified (as in this example), the API will return only persons with a similarity score above 0.7.

The SearchMode when using SearchMode.ACCURATE performs double the computation (and hence is slower) but can give a better score.

Usually omitting these parameters and letting the system use the defaults is good enough for most purposes.

from opencv.fr.search.schemas import SearchRequest, SearchMode
from pathlib import Path

# As before, you can also use OpenCV or Pillow images
image_base_path = Path("sample_images")
image_path = image_base_path / "varun.jpg"

search_request = SearchRequest([image_path])

If you want to recognize persons with masks, you can try a min_score of 0.66

Now that we have created a search_request, let's send it to the server:

result = sdk.search.search(search_request)

The result is simply a list of PersonSearchResult with each object containing a person attribute and a score attribute.

As an example, we can print the person and score attributes from the first item in the result:

print(result[0].person)
print(result[0].score)

A search can return more than one matched person, thus we use the first item from the result in the example above.

Sending a multi-face detection and search request

The search API we used before assumes that there is only one prominent face in an image. But what if we wanted to recognize all the people in an image?

This would mean we first need to detect all faces, and then search for each detected face, and each detected face could match with more than one persons.

To make this easy for you, we combined the search and detect functionalities in one API call.

As a user, you can choose to only detect faces, or to detect faces, and also search for them:

from opencv.fr.search.schemas import DetectionRequest, SearchOptions
from pathlib import Path

image_base_path = Path("sample_images")
image_path = image_base_path / "varun.jpg"

options = SearchOptions(
    min_score = 0.6,
    search_mode = "FAST"
)

detect_request_without_search = DetectionRequest(image_path)
detect_request_with_search = DetectionRequest(image_path, options)

In the example above, we construct two different detection requests.

For detect_request_without_search we do not specify any search options in the constructor.

For detect_request_with_search we specify search options.

When we call the API with detect_request_without_search, it will not search for the detected faces and return only the bounding boxes, landmarks, and thumbnails for all the detected faces:

print(sdk.search.detect(detect_object_without_search))

The following result shows one face was detected and gives us the position of the left (x ordinate), top (y ordinate), right (x ordinate), and the bottom (y ordinate) that specifies the bounding box of detection.

Then, it includes the position of Facial Landmarks (left_eye, right_eye, etc.) as [x, y] coordinates.

It further includes a thumbnail for the cropped face from the detection.

Finally, it includes a detection score, and a list of PersonSearchResult items that match the detection.

Had there been more detections, the top level list would have more such DetectionResponseItem objects.

Since we did not specify any SearchOptions, the list of person items will always be empty.

[
    {
        'box': {'left': 178, 'top': 139, 'right': 319, 'bottom': 314}, 
        'landmarks': {
            'left_eye': [214, 201], 
            'right_eye': [277, 203], 
            'nose': [242, 237], 
            'left_mouth': [211, 257], 
            'right_mouth': [276, 259]
        }, 
        'thumbnail': <PIL.Image.Image image mode=RGB size=112x112 at 0x7F9AB7E2BD30>, 
        'detection_score': 0.78, 
        'persons': []
    }
]

Now we can try our request where we did specify SearchOptions

print(sdk.search.detect(detect_request_with_search))

[
    {
        'box': {'left': 178, 'top': 139, 'right': 319, 'bottom': 314}, 
        'landmarks': {
            'left_eye': [214, 201], 
            'right_eye': [277, 203], 
            'nose': [242, 237], 
            'left_mouth': [211, 257], 
            'right_mouth': [276, 259]
        }, 
        'thumbnail': <PIL.Image.Image image mode=RGB size=112x112 at 0x7FE5E817B580>, 
        'detection_score': 0.78, 
        'persons': [
            {
                'person': {
                    'id': 'bb106660-85cd-44d0-8b80-2dc288503890', 
                    'name': 'Varun Chatterji', 
                    'images': [<PIL.Image.Image image mode=RGB size=112x112 at 0x7FE577712B80>], 
                    'gender': None, 
                    'collections': [], 
                    'date_of_birth': None, 
                    'nationality': None, 
                    'notes': None, 
                    'create_date': '2022-07-13 13:33:17.133421', 
                    'modified_date': '2022-07-13 13:33:39.689646'
                }, 
                'score': 0.9973
            }
        ]
    }
]

It is to be noted that more than one person can be associated with each detection.

Structure of the detection response

The detection response is a list of DetectionResponseItem objects. Each DetectionResponseItem contains:

  1. box - a box object of type Box, with object attributes left, top, right, left of type int
  2. landmarks - a landmarks object of type Landmarks, with attributes left_eye, right_eye, etc. each of type Coordinate.
    1. A Coordinate object has attributes x and y.
  3. thumbnail - a thumbnail of the detected face of type PIL.Image.Image
  4. detection_score - a detection score of type float
  5. persons - a list of persons matching the detection thumbnail of type PersonSearchResult

Verifying a specific Person

If you know the id of a specific person, and you would just like to know if a given image matches the person with the specified id, then this API is best suited to your needs.

As usual, we first create a VerificationRequest

from opencv.fr.search.schemas import VerificationRequest
from pathlib import Path

image_base_path = Path("sample_images")
image_path = image_base_path / "varun.jpg"

person_id = "bb106660-85cd-44d0-8b80-2dc288503890"
verification_request = VerificationRequest(person_id, [image_path])

Then we call the API:

print(sdk.search.verify(verification_request))

The response is an object of type PersonSearchResult and contains a person attribute of type Person and a score attribute of type float.

If the image doesn't match the id, then person will be None, and the score shall be 0.