28. Application exercise: version 10
28.1. Introduction
In the examples of clients from the tax calculation server, the threads sent N requests sequentially if they had to process N taxpayers. The idea here is to send a single request encapsulating the N taxpayers. For each of them, the [marié, enfants, salaire] information must be sent. This can be sent as parameters:
- of the URL. This results in a long, largely meaningless URL;
- in the body of the HTTP request. We know that this body is hidden from the user using a browser;
In both cases, we can use a request [GET] or [POST]. We will use a request POST with the parameters encapsulated in the body of the request HTTP.
The client/server architecture has not changed:
28.2. The web server
The [http-servers/05] file is initially obtained by copying the [http-servers/02] file. We return to the jSON exchanges between the client and the server. We have seen that switching from jSON to XML is very simple.
28.2.1. Configuration
The configuration for [config, config_database, config_layers] remains the same as in previous versions. We will not revisit it here.
28.2.2. The main script [main]
The [main] script is identical to the one in the [http-servers/02] folder that we copied. There is only one difference:
# Home URL
@app.route('/', methods=['POST'])
@auth.login_required
def index():
…
- line 2: now the URL is obtained via a POST;
28.2.3. The [index_controller] controller
The [index_controller] controller evolves as follows:
# import dependencies
import json
from flask_api import status
from werkzeug.local import LocalProxy
def execute(request: LocalProxy, config: dict) -> tuple:
# dependencies
from ImpôtsError import ImpôtsError
from TaxPayer import TaxPayer
# retrieve the body of the post - wait for a list of dictionaries
msg_erreur = None
list_dict_taxpayers = None
# the jSON body of POST
request_text = request.data
try:
# which we transform into a list of dictionaries
list_dict_taxpayers = json.loads(request_text)
except BaseException as erreur:
# we note the error
msg_erreur = f"le corps du POST n'est pas une chaîne jSON valide : {erreur}"
# do we have a non-empty list?
if not msg_erreur and (not isinstance(list_dict_taxpayers, list) or len(list_dict_taxpayers) == 0):
# we note the error
msg_erreur = "le corps du POST n'est pas une liste ou alors cette liste est vide"
# do we have a list of dictionaries?
if not msg_erreur:
erreur = False
i = 0
while not erreur and i < len(list_dict_taxpayers):
erreur = not isinstance(list_dict_taxpayers[i], dict)
i += 1
# mistake?
if erreur:
msg_erreur = "le corps du POST doit être une liste de dictionnaires"
# mistake?
if msg_erreur:
# an error response is sent to the client
résultats = {"réponse": {"erreurs": [msg_erreur]}}
return résultats, status.HTTP_400_BAD_REQUEST
# check TaxPayers one by one
# initially no errors
list_erreurs = []
for dict_taxpayer in list_dict_taxpayers:
# we create a TaxPayer from dict_taxpayer
msg_erreur = None
try:
# the following operation will eliminate cases where the parameters are not
# properties of the TaxPayer class as well as the cases where their values
# are incorrect
TaxPayer().fromdict(dict_taxpayer)
except BaseException as erreur:
msg_erreur = f"{erreur}"
# certain keys must be present in the dictionary
if not msg_erreur:
# the [marié, enfants, salaire] keys must be present in the dictionary
keys = dict_taxpayer.keys()
if 'marié' not in keys or 'enfants' not in keys or 'salaire' not in keys:
msg_erreur = "le dictionnaire doit inclure les clés [marié, enfants, salaire]"
# mistakes?
if msg_erreur:
# we note the error in the TaxPayer itself
dict_taxpayer['erreur'] = msg_erreur
# add TaxPayer to the error list
list_erreurs.append(dict_taxpayer)
# we've processed all the taxpayers - are there any mistakes?
if list_erreurs:
# an error response is sent to the client
résultats = {"réponse": {"erreurs": list_erreurs}}
return résultats, status.HTTP_400_BAD_REQUEST
# no mistakes, we can work
# data recovery from tax authorities
admindata = config["admindata"]
métier = config["layers"]["métier"]
try:
# process the TaxPayer one by one
list_taxpayers = []
for dict_taxpayer in list_dict_taxpayers:
# tAX CALCULATION
taxpayer = TaxPayer().fromdict(
{'marié': dict_taxpayer['marié'], 'enfants': dict_taxpayer['enfants'],
'salaire': dict_taxpayer['salaire']})
métier.calculate_tax(taxpayer, admindata)
# the result is stored as a dictionary
list_taxpayers.append(taxpayer.asdict())
# we send the response to the client
return {"réponse": {"results": list_taxpayers}}, status.HTTP_200_OK
except ImpôtsError as erreur:
# an error response is sent to the client
return {"réponse": {"erreurs": f"[{erreur}]"}}, status.HTTP_500_INTERNAL_SERVER_ERROR
- line 9: the controller receives:
- the client's [request] request;
- the server configuration [config];
- Lines 14–18: We retrieve the body of POST. The parameters embedded in the body of the HTTP request can be encoded in various ways. We have already encountered one such encoding: [x-www-form-urlencoded]. Here, we will use another encoding: jSON;
- line 18: [request.data] retrieves the body of the HTTP request. Here we retrieve text, and we know that this text is jSON, which represents a list of dictionaries [marié, enfants, salaire];
- lines 19–24: we retrieve this list of dictionaries;
- lines 22–24: if retrieving jSON failed, we log the error;
- lines 26–28: if we find that the retrieved object is not a list or is an empty list, we log the error;
- lines 29–38: if a list was successfully retrieved, we verify that it is indeed a list of dictionaries;
- lines 40–43: if an error occurred, we stop there and send an error response to the client;
- lines 45–69: we now check each of the dictionaries:
- they must contain the keys [marié, enfants, salaire];
- they must allow us to construct a valid [TaxPayer] object;
- lines 65–69: if an error was detected in a dictionary, it is added to that same dictionary under the key ‘error’;
- lines 72–75: the dictionaries containing errors have been added to the list [list_erreurs]. If this list is not empty, then it is sent in an error response to the client;
- line 77: at this point, we know that we can create a list of objects of type [TaxPayer] from the body of the request sent by the client;
- lines 84–91: we process the list of received dictionaries;
- line 86: from a dictionary, we create a [TaxPayer] object;
- line 89: we calculate the tax for this [TaxPayer];
- line 91: we know that [taxpayer] has been modified by the tax calculation. We convert it into a dictionary and add it to a list of results;
- line 93: we send this list of results to the client;
28.2.4. Server Testing
We will test the server with a Postman client:
- we launch the web server, SGBD, and the mail server, [hMailServer];
- We launch the Postman client and its console (Ctrl-Alt-C);
- In [1]: we send a request to [POST];
- in [2]: the server’s URL;
- in [3]: the body of the HTTP request;
- to [5]: we specify that this body must be sent as a string jSON;
- in [4]: we switch to [raw] mode to be able to copy/paste a string jSON;
- in [6]: paste the string jSON taken from one of the [résultats.json] files of the different versions. Then, for each taxpayer, keep only the [marié, salaire, enfants] properties;
- in [7], we look at the HTTP headers that the Postman client will send to the server;
- in [8], we see that it will send a [Content-Type] header indicating that the request contains a body encoded in jSON. This stems from the [5] choice made earlier;
- In [9-12]: we include the identifiers expected by the server in the request;
We send this request. The server’s response is as follows:
- in [3], we received jSON;
- in [4], the taxpayers' tax;
Let’s examine the client/server dialogue that took place in the Postman console (Ctrl-Alt-C):
The Postman client sent the following text:
- line 1: the POST to the server;
- line 2: the HTTP authentication header;
- line 3: the client tells the server that it is sending a jSON string and that this string is 824 bytes long (line 11);
- lines 13–69: the jSON body of the request;
The server responded with the following text:
HTTP/1.0 200 OK
Content-Type: application/json; charset=utf-8
Content-Length: 1461
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:16:34 GMT
{"réponse": {"results": [{"marié": "oui", "enfants": 2, "salaire": 55555, "impôt": 2814, "surcôte": 0, "taux": 0.14, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 2, "salaire": 50000, "impôt": 1384, "surcôte": 0, "taux": 0.14, "décôte": 384, "réduction": 347}, {"marié": "oui", "enfants": 3, "salaire": 50000, "impôt": 0, "surcôte": 0, "taux": 0.14, "décôte": 720, "réduction": 0}, {"marié": "non", "enfants": 2, "salaire": 100000, "impôt": 19884, "surcôte": 4480, "taux": 0.41, "décôte": 0, "réduction": 0}, {"marié": "non", "enfants": 3, "salaire": 100000, "impôt": 16782, "surcôte": 7176, "taux": 0.41, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 3, "salaire": 100000, "impôt": 9200, "surcôte": 2180, "taux": 0.3, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 5, "salaire": 100000, "impôt": 4230, "surcôte": 0, "taux": 0.14, "décôte": 0, "réduction": 0}, {"marié": "non", "enfants": 0, "salaire": 100000, "impôt": 22986, "surcôte": 0, "taux": 0.41, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 2, "salaire": 30000, "impôt": 0, "surcôte": 0, "taux": 0.0, "décôte": 0, "réduction": 0}, {"marié": "non", "enfants": 0, "salaire": 200000, "impôt": 64210, "surcôte": 7498, "taux": 0.45, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 3, "salaire": 200000, "impôt": 42842, "surcôte": 17283, "taux": 0.41, "décôte": 0, "réduction": 0}]}}
- line 1: the request was successful;
- line 2: the body of the server's response is a string jSON. It is 1461 bytes long (line 3);
- line 7: the server's response jSON;
Now let’s test some error cases.
Case 1: we send anything
POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 47652706-9744-46a0-a682-de010e5406c0
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 3
abc
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 125
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:43:27 GMT
{"réponse": {"erreurs": ["le corps du POST n'est pas une chaîne jSON valide : Expecting value: line 1 column 1 (char 0)"]}}
- line 13: the string [abc] was sent, which is not a valid jSON string (line 3);
- line 15: the server responds with a 400 error code;
- line 21: the server’s response jSON;
Case 2: Let’s send a valid string jSON that is not a list
POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 03b64735-9239-47b3-b92d-be7c9ebc7559
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 17
{"att1":"value1"}
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 97
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:50:11 GMT
{"réponse": {"erreurs": ["le corps du POST n'est pas une liste ou alors cette liste est vide"]}}
Case 3: Let’s send a string jSON that is a list whose elements are not all dictionaries
POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: a1528a5f-777c-413f-b3be-7d4e9955b12a
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 7
[0,1,2]
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 85
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:52:10 GMT
{"réponse": {"erreurs": ["le corps du POST doit être une liste de dictionnaires"]}}
Case 4: Let’s send a list of dictionaries with a dictionary that doesn’t have the correct keys
POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: ba964d81-c9d9-46ff-a521-b4c4e5639484
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 19
[{"att1":"value1"}]
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 112
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:54:33 GMT
{"réponse": {"erreurs": [{"att1": "value1", "erreur": "MyException[2, la clé [att1] n'est pas autorisée]"}]}}
Case 5: Let’s send a list of dictionaries with a dictionary containing missing keys:
POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 98aec51d-f37d-4c14-81cd-c7ffcbbcdc65
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 18
[{"marié":"oui"}]
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 125
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:56:40 GMT
{"réponse": {"erreurs": [{"marié": "oui", "erreur": "le dictionnaire doit inclure les clés [marié, enfants, salaire]"}]}}
Case 6: Let’s send a list of dictionaries with one dictionary containing the correct keys but some with incorrect values:
POST / HTTP/1.1
Authorization: Basic YWRtaW46YWRtaW4=
Content-Type: application/json
User-Agent: PostmanRuntime/7.26.2
Accept: */*
Cache-Control: no-cache
Postman-Token: 3083e601-dee4-4e15-9ea4-fc0328d0fcf0
Host: localhost:5000
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
Content-Length: 46
[{"marié":"x", "enfants":"x", "salaire":"x"}]
HTTP/1.0 400 BAD REQUEST
Content-Type: application/json; charset=utf-8
Content-Length: 167
Server: Werkzeug/1.0.1 Python/3.8.1
Date: Tue, 28 Jul 2020 07:59:32 GMT
{"réponse": {"erreurs": [{"marié": "x", "enfants": "x", "salaire": "x", "erreur": "MyException[31, l'attribut marié [x] doit avoir l'une des valeurs oui / non]"}]}}
28.3. The web client
The file [http-clients/05] (version 10) is initially obtained by copying the file [http-clients/02] (version 7). It is then modified.
28.3.1. The [dao] layer
The layer [dao] is implemented by the following class [ImpôtsDaoWithHttpClient]:
# imports
import requests
from flask_api import status
from AbstractImpôtsDao import AbstractImpôtsDao
from AdminData import AdminData
from ImpôtsError import ImpôtsError
from InterfaceImpôtsMétier import InterfaceImpôtsMétier
from TaxPayer import TaxPayer
class ImpôtsDaoWithHttpClient(AbstractImpôtsDao, InterfaceImpôtsMétier):
# manufacturer
def __init__(self, config: dict):
…
# unused method
def get_admindata(self) -> AdminData:
pass
# tAX CALCULATION
def calculate_tax(self, taxpayer: TaxPayer, admindata: AdminData = None):
…
# bulk tax calculation
def calculate_tax_in_bulk_mode(self, taxpayers: list) -> list:
# we let the exceptions rise
# transform taxpayers into a list of dictionaries
# we keep only the [marié, enfants, salaire] properties
list_dict_taxpayers = list(
map(lambda taxpayer:
taxpayer.asdict(included_keys=[
'_TaxPayer__marié',
'_TaxPayer__enfants',
'_TaxPayer__salaire']),
taxpayers))
# server connection
config_server = self.__config_server
if config_server['authBasic']:
response = requests.post(config_server['urlServer'], json=list_dict_taxpayers,
auth=(config_server["user"]["login"],
config_server["user"]["password"]))
else:
response = requests.post(config_server['urlServer'], json=list_dict_taxpayers)
# mode debug ?
if self.__debug:
# logger
if not self.__logger:
self.__logger = self.__config['logger']
# log on
self.__logger.write(f"{response.text}\n")
# response status code HTTP
status_code = response.status_code
# we put the response jSON in a dictionary
résultat = response.json()
# error if status code other than 200 OK
if status_code != status.HTTP_200_OK:
# we know that the errors were associated with the [erreurs] key in the response
raise ImpôtsError(93, résultat['réponse']['erreurs'])
# we know that the result has been associated with the [results] key in the response
list_dict_taxpayers2 = résultat['réponse']['results']
# the initial list of taxpayers is updated with the results received
for i in range(len(taxpayers)):
# update of taxpayers[i]
taxpayers[i].fromdict(list_dict_taxpayers2[i])
# here the [taxpayers] parameter has been updated with the server results
- lines 1–26: the code remains the same as in version 7 and other versions;
- lines 27–70: a new method, [calculate_tax_in_bulk_mode], is introduced to calculate the tax for a list of taxpayers;
- line 28: [taxpayers] is this list of taxpayers;
- lines 31–39: we convert a list of objects of type [TaxPayer] into a list of dictionaries using a function [map];
- lines 34–38: the lambda function used transforms an object of type [TaxPayer] into a dictionary of type [dict] with only the keys [marié, enfants, salaire]. To do this, we use the parameter named [included_keys] from the [BaseEntity.asdict] method. Note that to determine the exact names of the properties to include in the [excluded_keys, included_keys] parameters, you must use the predefined dictionary [taxpayer.__dict__];
- lines 41–48: connect to the server and retrieve its response HTTP;
- lines 44, 48:
- the static method [requests.post] is used to send a POST to the server;
- the parameter named [json] is used to indicate that the body of POST is a string jSON. This will have two consequences:
- the object assigned to the parameter named [json], in this case a list of dictionaries, will be converted into a string jSON;
- the header
will be included in the headers of HTTP from POST;
- line 59: the server’s response jSON is deserialized into the dictionary [résultat];
- lines 61–63: any error sent by the server is handled;
- line 65: the results of the tax calculation are in a list of dictionaries;
- lines 67–69: these results are used to update the initial list of taxpayers [taxpayers] originally received by the method on line 28;
- line 70: here, the initial list of taxpayers has been updated with the tax calculation results;
28.3.2. The main script [main]
The main script [main] evolves as follows: only the function [thread_function] executed by the threads created by the client is modified. The rest of the code remains unchanged.
# execution of the [dao] layer in a thread
# taxpayers is a list of taxpayers
def thread_function(dao: ImpôtsDaoWithHttpClient, logger: Logger, taxpayers: list):
# log thread start
thread_name = threading.current_thread().name
nb_taxpayers = len(taxpayers)
# log
logger.write(f"début du calcul de l'impôt des {nb_taxpayers} contribuables\n")
# taxpayers' taxes are calculated
dao.calculate_tax_in_bulk_mode(taxpayers)
# log
logger.write(f"fin du calcul de l'impôt des {nb_taxpayers} contribuables\n")
- lines 9-10: whereas previously we had a loop that successively passed each taxpayer to the [dao.calculate_tax] method, here we make a single call to the [dao.calculate_tax_in_bulk_mode] method, passing all taxpayers to it;
28.3.3. Client Execution
We will compare the execution times of versions:
- 7, where each taxpayer is the subject of a HTTP query;
- 10 (this one), where taxpayers are grouped into a single HTTP query;
First, version 6. To compare the two versions, we set the server’s [sleep_time] property to zero so that there is no forced thread wait. The client logs are as follows:
2020-07-28 14:20:45.811347, Thread-1 : début du thread [Thread-1] avec 4 contribuable(s)
2020-07-28 14:20:45.811347, Thread-1 : début du calcul de l'impôt de {"id": 1, "marié": "oui", "enfants": 2, "salaire": 55555}
…
2020-07-28 14:20:45.913065, Thread-3 : fin du calcul de l'impôt de {"id": 11, "marié": "oui", "enfants": 3, "salaire": 200000, "impôt": 42842, "surcôte": 17283, "taux": 0.41, "décôte": 0, "réduction": 0}
2020-07-28 14:20:45.913065, Thread-3 : fin du thread [Thread-3]
The client execution time to calculate the tax for 11 taxpayers is therefore [913065-811347= 101718], i.e., approximately 102 milliseconds.
Let’s do the same with version 10 (sleep_time from the server at zero). The client logs are then as follows:
2020-07-28 14:25:31.871428, Thread-1 : début du calcul de l'impôt des 4 contribuables
2020-07-28 14:25:31.873594, Thread-2 : début du calcul de l'impôt des 3 contribuables
2020-07-28 14:25:31.877429, Thread-3 : début du calcul de l'impôt des 3 contribuables
2020-07-28 14:25:31.882855, Thread-4 : début du calcul de l'impôt des 1 contribuables
2020-07-28 14:25:31.930723, Thread-2 : {"réponse": {"results": [{"marié": "non", "enfants": 3, "salaire": 100000, "impôt": 16782, "surcôte": 7176, "taux": 0.41, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 3, "salaire": 100000, "impôt": 9200, "surcôte": 2180, "taux": 0.3, "décôte": 0, "réduction": 0}, {"marié": "oui", "enfants": 5, "salaire": 100000, "impôt": 4230, "surcôte": 0, "taux": 0.14, "décôte": 0, "réduction": 0}]}}
….
2020-07-28 14:25:31.935958, Thread-4 : fin du calcul de l'impôt des 1 contribuables
2020-07-28 14:25:31.935958, Thread-1 : fin du calcul de l'impôt des 4 contribuables
The client execution time to calculate the tax for 11 taxpayers is therefore [935958-871428= 64530 ns] (line 8 – line 1), i.e., approximately 65 milliseconds. This new version 10 thus results in a gain of approximately 57% over version 7.
28.3.4. Client-side [dao] layer tests
The [TestHttpClientDao] client test for the version 10 is very similar to that of the version 7:
import unittest
from Logger import Logger
class TestHttpClientDao(unittest.TestCase):
def test_1(self) -> None:
from TaxPayer import TaxPayer
# { 'married': 'yes', 'children': 2, 'salary': 55555,
# tax': 2814, 'surcôte': 0, 'décôte': 0, 'réduction': 0, 'taux': 0.14}
taxpayer = TaxPayer().fromdict({"marié": "oui", "enfants": 2, "salaire": 55555})
dao.calculate_tax_in_bulk_mode([taxpayer])
# check
self.assertAlmostEqual(taxpayer.impôt, 2815, delta=1)
self.assertEqual(taxpayer.décôte, 0)
self.assertEqual(taxpayer.réduction, 0)
self.assertAlmostEqual(taxpayer.taux, 0.14, delta=0.01)
self.assertEqual(taxpayer.surcôte, 0)
…
if __name__ == '__main__':
# configure the application
import config
config = config.configure({})
# logger
logger = Logger(config["logsFilename"])
# it is stored in the config
config["logger"] = logger
# retrieve the [dao] layer
dao = config["layers"]["dao"]
# test methods are executed
print("tests en cours...")
unittest.main()
- line 14: instead of calling the [dao.calculate_tax] method, we call the [dao.calculate_tax_in_bulk_mode] method, passing it a list (indicated by square brackets) of a taxpayer;
All tests pass.