28. Application exercise: version 10

28.1. Introduction

In the examples of clients for the tax calculation server, the threads sent N requests sequentially if they had to process N taxpayers. The idea here is to send a single request encapsulating the N taxpayers. For each of them, the information [married, children, salary] must be sent. These can be sent as parameters:

in the URL. This results in a long, meaningless URL;
in the body of the HTTP request. We know that this body is hidden from the user using a browser;

In both cases, you can use a [GET] or [POST] request. We will use a POST request with the parameters embedded in the HTTP request body.

The client/server architecture has not changed:

28.2. The web server

The [http-servers/05] folder is initially created by copying the [http-servers/02] folder. We return to JSON exchanges between the client and the server. We have seen that switching from JSON to XML is very simple.

28.2.1. Configuration

The configuration [config, config_database, config_layers] remains the same as in previous versions. We won’t go over it again.

28.2.2. The main script [main]

The [main] script is identical to the one in the [http-servers/02] folder that we copied. Only one thing differs:


# Home URL
@app.route('/', methods=['POST'])
@auth.login_required
def index():
    …

line 2: the / URL is now accessed via a POST request;

28.2.3. The [index_controller]

The [index_controller] evolves as follows:


# import dependencies

import json

from flask_api import status
from werkzeug.local import LocalProxy


def execute(request: LocalProxy, config: dict) -> tuple:
    # dependencies
    from TaxError import TaxError
    from TaxPayer import TaxPayer

    # retrieve the post body - expect a list of dictionaries
    error_msg = None
    list_dict_taxpayers = None
    # the JSON body of the POST
    request_text = request.data
    try:
        # which we convert into a list of dictionaries
        list_dict_taxpayers = json.loads(request_text)
    except BaseException as error:
        # log the error
        msg_error = f"The POST body is not a valid JSON string: {error}"
    # Is the list non-empty?
    if not error_message and (not isinstance(taxpayers_list_dict, list) or len(taxpayers_list_dict) == 0):
        # Log the error
        error_message = "The POST body is not a list, or the list is empty"
    # Do we have a list of dictionaries?
    if not error_message:
        error = False
        i = 0
        while not error and i < len(list_dict_taxpayers):
            error = not isinstance(list_dict_taxpayers[i], dict)
            i += 1
        # error?
        if error:
            error_message = "The body of the POST request must be a list of dictionaries"
    # error?
    if error_message:
        # send an error response to the client
        results = {"response": {"errors": [error_message]}}
        return results, status.HTTP_400_BAD_REQUEST

    # check the TaxPayers one by one
    # initially, no errors
    error_list = []
    for tax_payer_dict in tax_payer_dicts:
        # create a TaxPayer from dict_taxpayer
        error_message = None
        try:
            # The following operation will eliminate cases where the parameters are not
            # properties of the TaxPayer class, as well as cases where their values
            # are incorrect
            TaxPayer().fromdict(dict_taxpayer)
        except BaseException as error:
            error_message = f"{error}"
        # certain keys must be present in the dictionary
        if not error_message:
            # the keys [married, children, salary] must be present in the dictionary
            keys = dict_taxpayer.keys()
            if 'married' is not in keys or 'children' is not in keys or 'salary' is not in keys:
                msg_error = "The dictionary must include the keys [married, children, salary]"
        # Any errors?
        if error_message:
            # record the error in the TaxPayer itself
            dict_taxpayer['error'] = error_message
            # add the TaxPayer to the list of errors
            list_errors.append(dict_taxpayer)

    # Have all taxpayers been processed? Are there any errors?
    if list_errors:
        # send an error response to the client
        results = {"response": {"errors": list_errors}}
        return results, status.HTTP_400_BAD_REQUEST

    # No errors, we can proceed
    # retrieve data from the tax administration
    admindata = config["admindata"]
    business = config["layers"]["business"]
    try:
        # process taxpayers one by one
        list_taxpayers = []
        for dict_taxpayer in list_dict_taxpayers:
            # calculate the tax
            taxpayer = TaxPayer().fromdict(
                {'married': dict_taxpayer['married'], 'children': dict_taxpayer['children'],
                 'salary': dict_taxpayer['salary']})
            job.calculate_tax(taxpayer, admindata)
            # store the result as a dictionary
            list_taxpayers.append(taxpayer.asdict())
        # send the response to the client
        return {"response": {"results": list_taxpayers}}, status.HTTP_200_OK
    except ImpôtsError as error:
        # Send an error response to the client
        return {"response": {"errors": f"[{error}]"}}, status.HTTP_500_INTERNAL_SERVER_ERROR

Line 9: The controller receives:
lines 14–18: We retrieve the POST body. The parameters encapsulated in the HTTP request body can be encoded in various ways. We’ve already encountered one: [x-www-form-urlencoded]. Here, we’ll use another encoding: JSON;
line 18: [request.data] retrieves the body of the HTTP request. Here, we retrieve text, and we know that this text is JSON representing a list of dictionaries [married, children, salary];
lines 19–24: we retrieve this list of dictionaries;
lines 22–24: if the JSON retrieval failed, we log the error;
lines 26–28: if we find that the retrieved object is not a list or is an empty list, we log the error;
lines 29–38: if a list was successfully retrieved, verify that it is indeed a list of dictionaries;
lines 40–43: if an error occurred, we stop here and send an error response to the client;
lines 45–69: we now check each of the dictionaries:
- they must contain the keys [married, children, salary];
- they must allow us to construct a valid [TaxPayer] object;
lines 65–69: if an error is detected in a dictionary, it is added to that same dictionary under the key ‘error’;
lines 72–75: the dictionaries containing errors have been collected in the list [list_errors]. If this list is not empty, then it is sent in an error response to the client;
line 77: at this point, we know we can create a list of objects of type [TaxPayer] from the body of the request sent by the client;
lines 84–91: We process the list of received dictionaries;
line 86: from a dictionary, we create a [TaxPayer] object;
line 89: we calculate the tax for this [TaxPayer];
line 91: we know that [taxpayer] has been modified by the tax calculation. We convert it into a dictionary and add it to a list of results;
line 93: this list of results is sent to the client;

28.2.4. Server Testing

We will test the server using a Postman client:

We start the web server, the DBMS, and the mail server [hMailServer];
We launch the Postman client and its console (Ctrl-Alt-C);

in [1]: we send a [POST] request;
in [2]: the server’s URL;
in [3]: the body of the HTTP request;
in [5]: we specify that this body should be sent as a JSON string;
in [4]: we switch to [raw] mode to be able to copy and paste a JSON string;
in [6]: paste the JSON string taken from one of the [results.json] files for the different versions. Then, for each taxpayer, keep only the properties [married, salary, children];

in [7], we look at the HTTP headers that the Postman client will send to the server;
in [8], we see that it will send a [Content-Type] header indicating that the request contains a JSON-encoded body. This is due to the choice made in [5] earlier;

in [9-12]: we include the credentials expected by the server in the request;

We send this request. The server's response is as follows:

in [3], we received JSON;
in [4], the taxpayers' tax;

Let’s examine the client/server dialogue that took place in the Postman console (Ctrl-Alt-C):

The Postman client sent the following text:


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 03c4aa28-5a5d-4bb5-ac51-7ad51968c71d
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 824

[
  {
    "married": "yes",
    "children": 2,
    "salary": 55555
  },
  {
    "married": "yes",
    "children": 2,
    "salary": 50000
  },
  {
    "married": "yes",
    "children": 3,
    "salary": 50000
  },
  {
    "married": "no",
    "children": 2,
    "salary": 100,000
  },
  {
    "married": "no",
    "children": 3,
    "salary": 100,000
  },
  {
    "married": "yes",
    "children": 3,
    "salary": 100,000
  },
  {
    "married": "yes",
    "children": 5,
    "salary": 100,000
  },
  {
    "married": "no",
    "children": 0,
    "salary": 100,000
  },
  {
    "married": "yes",
    "children": 2,
    "salary": 30,000
  },
  {
    "married": "no",
    "children": 0,
    "salary": 200,000
  },
  {
    "married": "yes",
    "children": 3,
    "salary": 200,000
  }
]

line 1: the POST request to the server;
line 2: the HTTP authentication header;
line 3: the client tells the server that it is sending a JSON string and that this string is 824 bytes long (line 11);
lines 13–69: the JSON body of the request;

The server responded with the following text:


HTTP/1.0 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 1461
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, Jul 28, 2020 07:16:34 GMT

{"response": {"results": [{"married": "yes", "children": 2, "salary": 55555, "tax": 2814, "surcharge": 0, "rate": 0.14, "discount": 0, "reduction": 0}, {"married": "yes", "children": 2, "salary": 50000, "tax": 1384, "surcharge": 0, "rate": 0.14, "discount": 384, "reduction": 347}, {"married": "yes", "children": 3, "salary": 50000, "tax": 0, "surcharge": 0, "rate": 0.14, "discount": 720, "reduction": 0}, {"married": "no", "children": 2, "salary": 100000, "tax": 19,884, "surcharge": 4,480, "rate": 0.41, "discount": 0, "reduction": 0}, {"married": "no", "children": 3, "salary": 100,000, "tax": 16,782, "surcharge": 7,176, "rate": 0.41, "discount": 0, "reduction": 0}, {"married": "yes", "children": 3, "salary": 100,000, "tax": 9200, "surcharge": 2180, "rate": 0.3, "discount": 0, "reduction": 0}, {"married": "yes", "children": 5, "salary": 100000, "tax": 4230, "surcharge": 0, "rate": 0.14, "discount": 0, "reduction": 0}, {"married": "no", "children": 0, "salary": 100000, "tax": 22,986, "surcharge": 0, "rate": 0.41, "discount": 0, "reduction": 0}, {"married": "yes", "children": 2, "salary": 30000, "tax": 0, "surcharge": 0, "rate": 0.0, "discount": 0, "reduction": 0}, {"married": "no", "children": 0, "salary": 200000, "tax": 64210, "surcharge": 7498, "rate": 0.45, "discount": 0, "reduction": 0}, {"married": "yes", "children": 3, "salary": 200000, "tax": 42842, "surcharge": 17283, "rate": 0.41, "discount": 0, "reduction": 0}]}}

line 1: the request was successful;
line 2: the body of the server's response is a JSON string. It is 1461 bytes long (line 3);
line 7: the server's JSON response;

Now let’s test some error cases.

Case 1: we send anything


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 47652706-9744-46a0-a682-de010e5406c0
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 3

abc

HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 125
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:43:27 GMT

{"response": {"errors": ["the POST body is not a valid JSON string: Expecting value: line 1 column 1 (char 0)"]}}

line 13: the string [abc] was sent, which is not a valid JSON string (line 3);
line 15: the server responds with a 400 error code;
line 21: the server's JSON response;

Case 2: Let’s send a valid JSON string that is not a list


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 03b64735-9239-47b3-b92d-be7c9ebc7559
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 17

{"att1":"value1"}

HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 97
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, Jul 28, 2020 7:50:11 AM GMT

{"response": {"errors": ["the POST body is not a list or the list is empty"]}}

Case 3: Let's send a JSON string that is a list whose elements are not all dictionaries


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: a1528a5f-777c-413f-b3be-7d4e9955b12a
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 7

[0,1,2]

HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 85
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, Jul 28, 2020 07:52:10 GMT

{"response": {"errors": ["the POST body must be a list of dictionaries"]}}

Case 4: Let’s send a list of dictionaries with a dictionary that doesn’t have the correct keys


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: ba964d81-c9d9-46ff-a521-b4c4e5639484
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 19

[{"att1":"value1"}]

HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 112
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, Jul 28, 2020 7:54:33 AM GMT

{"response": {"errors": [{"att1": "value1", "error": "MyException[2, the key [att1] is not allowed]"}]}}

Case 5: Let's send a list of dictionaries with a dictionary containing missing keys:


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 98aec51d-f37d-4c14-81cd-c7ffcbbcdc65
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 18

[{"married":"yes"}]

HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 125
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, Jul 28, 2020 7:56:40 AM GMT

{"response": {"errors": [{"married": "yes", "error": "the dictionary must include the keys [married, children, salary]"}]}}

Case 6: Let’s send a list of dictionaries with one dictionary containing the correct keys but some with incorrect values:


POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 3083e601-dee4-4e15-9ea4-fc0328d0fcf0
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 46

[{"married":"x", "children":"x", "salary":"x"}]

HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 167
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:59:32 GMT

{"response": {"errors": [{"married": "x", "children": "x", "salary": "x", "error": "MyException[31, the married attribute [x] must have one of the values yes / no]"}]}}

28.3. The web client

The [http-clients/05] file (version 10) is initially obtained by copying the [http-clients/02] file (version 7). It is then modified.

28.3.1. The [dao] layer

The [dao] layer is implemented by the following [ImpôtsDaoWithHttpClient] class:


# imports

import requests
from flask_api import status

from AbstractTaxDao import AbstractTaxDao
from AdminData import AdminData
from TaxError import TaxError
from BusinessTaxInterface import BusinessTaxInterface
from TaxPayer import TaxPayer


class TaxDaoWithHttpClient(AbstractTaxDao, BusinessTaxInterface):

    # constructor
    def __init__(self, config: dict):
        …

    # unused method
    def get_admindata(self) -> AdminData:
        pass

    # tax calculation
    def calculate_tax(self, taxpayer: TaxPayer, admindata: AdminData = None):
        …

    # Calculate tax in bulk mode
    def calculate_tax_in_bulk_mode(self, taxpayers: list) -> list:
        # let exceptions be raised

        # convert taxpayers into a list of dictionaries
        # keep only the [married, children, salary] properties
        list_dict_taxpayers = list(
            map(lambda taxpayer:
                taxpayer.asdict(included_keys=[
                    '_TaxPayer__married',
                    '_TaxPayer__children',
                    '_TaxPayer__salary']),
                taxpayers))

        # Connect to the server
        config_server = self.__config_server
        if config_server['authBasic']:
            response = requests.post(config_server['urlServer'], json=list_dict_taxpayers,
                                     auth = (config_server["user"]["login"],
                                           config_server["user"]["password"]))
        else:
            response = requests.post(config_server['urlServer'], json=list_dict_taxpayers)
        # debug mode?
        if self.__debug:
            # logger
            if not self.__logger:
                self.__logger = self.__config['logger']
            # logging
            self.__logger.write(f"{response.text}\n")
        # HTTP response status code
        status_code = response.status_code
        # put the JSON response into a dictionary
        result = response.json()
        # error if status code is not 200 OK
        if status_code != status.HTTP_200_OK:
            # we know that errors have been associated with the [errors] key in the response
            raise ImpôtsError(93, result['response']['errors'])
        # we know that the result has been associated with the [results] key in the response
        list_dict_taxpayers2 = result['response']['results']
        # update the initial list of taxpayers with the received results
        for i in range(len(taxpayers)):
            # Update taxpayers[i]
            taxpayers[i].fromdict(list_dict_taxpayers2[i])
        # Here, the [taxpayers] parameter has been updated with the results from the server

lines 1–26: the code remains the same as in version 7 and other versions;
lines 27–70: a new method [calculate_tax_in_bulk_mode] is introduced, whose purpose is to calculate the tax for a list of taxpayers;
line 28: [taxpayers] is this list of taxpayers;
lines 31–39: we convert a list of [TaxPayer] objects into a list of dictionaries using the [map] function;
lines 34–38: the lambda function used transforms an object of type [TaxPayer] into a dictionary of type [dict] with only the keys [married, children, salary]. To do this, we use the parameter named [included_keys] from the [BaseEntity.asdict] method. Note that to determine the exact names of the properties to include in the [excluded_keys, included_keys] parameters, you must use the predefined dictionary [taxpayer.__dict__];
lines 41–48: connect to the server and retrieve its HTTP response;
lines 44, 48:
- We use the static method [requests.post] to send a POST request to the server;
- the parameter named [json] is used to indicate that the POST body is a JSON string. This will have two consequences:
  - the object assigned to the named parameter [json], in this case a list of dictionaries, will be converted to a JSON string;
  - the header

Content-Type: application/json

will be included in the POST’s HTTP headers;

line 59: the server’s JSON response is deserialized into the [result] dictionary;
lines 61–63: any error sent by the server is handled;
line 65: the tax calculation results are in a list of dictionaries;
lines 67–69: these results are used to update the initial list of taxpayers [taxpayers] originally received by the method on line 28;
line 70: here, the initial list of taxpayers has been updated with the tax calculation results;

28.3.2. The main script [main]

The main script [main] evolves as follows: only the function [thread_function] executed by the threads created by the client is modified. The rest of the code remains unchanged.


# execution of the [dao] layer in a thread
# taxpayers is a list of taxpayers
def thread_function(dao: ImpôtsDaoWithHttpClient, logger: Logger, taxpayers: list):
    # log start of thread
    thread_name = threading.current_thread().name
    nb_taxpayers = len(taxpayers)
    # log
    logger.write(f"Starting tax calculation for {nb_taxpayers} taxpayers\n")
    # Calculate the tax for the taxpayers
    dao.calculate_tax_in_bulk_mode(taxpayers)
    # log
    logger.write(f"End of tax calculation for {nb_taxpayers} taxpayers\n")

lines 9–10: whereas previously we had a loop that passed each taxpayer in turn to the [dao.calculate_tax] method, here we make a single call to the [dao.calculate_tax_in_bulk_mode] method, passing all taxpayers to it;

28.3.3. Client Execution

We will compare the execution times of versions:

7, where each taxpayer is the subject of an HTTP request;
10 (this one), where taxpayers are grouped into a single HTTP request;

First, version 6. To compare the two versions, we set the server’s [sleep_time] property to zero so that there is no forced waiting for threads. The client logs are as follows:


2020-07-28 14:20:45.811347, Thread-1: start of thread [Thread-1] with 4 taxpayer(s)
2020-07-28 14:20:45.811347, Thread-1: Start of tax calculation for {"id": 1, "married": "yes", "children": 2, "salary": 55555}
…
2020-07-28 14:20:45.913065, Thread-3: End of tax calculation for {"id": 11, "married": "yes", "children": 3, "salary": 200000, "tax": 42842, "surcharge": 17283, "rate": 0.41, "discount": 0, "reduction": 0}
2020-07-28 14:20:45.913065, Thread-3: end of thread [Thread-3]

The client's execution time to calculate the tax for 11 taxpayers is therefore [913065-811347= 101718], i.e., approximately 102 milliseconds.

Let’s do the same with version 10 (server sleep_time set to zero). The client logs are then as follows:


2020-07-28 14:25:31.871428, Thread-1: start of tax calculation for the 4 taxpayers
2020-07-28 14:25:31.873594, Thread-2: start of tax calculation for the 3 taxpayers
2020-07-28 14:25:31.877429, Thread-3: Start of tax calculation for 3 taxpayers
2020-07-28 14:25:31.882855, Thread-4: Start of tax calculation for 1 taxpayer
2020-07-28 14:25:31.930723, Thread-2 : {"response": {"results": [{"married": "no", "children": 3, "salary": 100000, "tax": 16,782, "surcharge": 7,176, "rate": 0.41, "discount": 0, "reduction": 0}, {"married": "yes", "children": 3, "salary": 100,000, "tax": 9200, "surcharge": 2180, "rate": 0.3, "discount": 0, "reduction": 0}, {"married": "yes", "children": 5, "salary": 100000, "tax": 4230, "surcharge": 0, "rate": 0.14, "discount": 0, "reduction": 0}]}}
….
2020-07-28 14:25:31.935958, Thread-4: End of tax calculation for 1 taxpayer
2020-07-28 14:25:31.935958, Thread-1: End of tax calculation for 4 taxpayers

The client’s execution time to calculate the tax for 11 taxpayers is therefore [935958-871428= 64530 ns] (line 8 – line 1), i.e., approximately 65 milliseconds. This new version 10 thus delivers a performance gain of approximately 57% over version 7.

28.3.4. Tests of the client’s [dao] layer

The [TestHttpClientDao] test for the client in version 10 is very similar to that in version 7:


import unittest

from Logger import Logger


class TestHttpClientDao(unittest.TestCase):

    def test_1(self) -> None:
        from TaxPayer import TaxPayer

        # {'married': 'yes', 'children': 2, 'salary': 55555,
        # 'tax': 2814, 'surcharge': 0, 'discount': 0, 'reduction': 0, 'rate': 0.14}
        taxpayer = TaxPayer().fromdict({"married": "yes", "children": 2, "salary": 55555})
        dao.calculate_tax_in_bulk_mode([taxpayer])
        # verification
        self.assertAlmostEqual(taxpayer.tax, 2815, delta=1)
        self.assertEqual(taxpayer.discount, 0)
        self.assertEqual(taxpayer.reduction, 0)
        self.assertAlmostEqual(taxpayer.rate, 0.14, delta=0.01)
        self.assertEqual(taxpayer.surcharge, 0)

    …

if __name__ == '__main__':
    # configure the application
    import config
    config = config.configure({})

    # logger
    logger = Logger(config["logsFilename"])
    # Store it in the config
    config["logger"] = logger
    # retrieve the [dao] layer
    dao = config["layers"]["dao"]

    # run the test methods
    print("Tests in progress...")
    unittest.main()

line 14: instead of calling the [dao.calculate_tax] method, we call the [dao.calculate_tax_in_bulk_mode] method, passing it a list (indicated by the square brackets) of taxpayers;

All tests pass.