5. Solving the Three Problems with ChatGPT

5.1. Introduction

Here is a first screenshot of a ChatGPT session:

In [1-3], the three problems posed to ChatGPT;
In [4], the ChatGPT URL;
In [5], the version of ChatGPT used;

ChatGPT is a product of OpenAI available at the URL [https://chatgpt.com/]. To view a history of your question-and-answer sessions like the one above, you need to create an account. Furthermore, like all other AIs tested, ChatGPT limits the number of questions you can ask and the number of files you can upload. When this limit is reached, the session ends, and you are offered the option to continue later. The limits imposed by ChatGPT are reached very quickly. To create this tutorial, I had to purchase a one-month paid subscription.

The ChatGPT interface is as follows:

At [1], to attach files to the question asked;
At [2], the question asked;
At [3], to run the AI;

5.2. Problem 1

The question for ChatGPT:

ChatGPT responds correctly.

5.3. Problem 2

This involves calculating the tax using the PDF. To be honest, we’ll use the PDF generated by Gemini, which corrects errors in the original PDF.

In [1], we provided the PDF generated by Gemini;
In [2], we added the unit test through which Gemini demonstrated its superiority:

1	`test12 : (2, 2, 49500) -> (1297, 431, 324)`

We run ChatGPT. It takes about 3 minutes to generate its response. Unlike Gemini, it does provide a working link to retrieve the generated script. We load this into PyCharm:

The [chatGPT1] script works on the first try. There’s no contest here; on this problem, ChatGPT outperformed Gemini.

The script [chatGPT1] provided by ChatGPT is as follows:

#  -*- coding: utf-8 -*-
"""
Calcul de l'impôt 2019 (revenus 2018) en Python.

Hypothèses et règles (d'après le PDF fourni) :
- Tous les revenus sont portés sur le déclarant 1 (même pour un couple).
- Abattement forfaitaire de 10 % encadré par un minimum et un maximum.
- Barème progressif 2018 appliqué au quotient familial (QF).
- Plafonnement du quotient familial : avantage par demi-part enfant limité.
- Décote et réduction de 20 % pour faibles revenus, avec la règle de repêchage.
- Ordre des correctifs : plafonnement QF -> décote -> réduction 20 %.
- Arrondis : abattement (↑), décote (↑), réduction (↑), impôt final (↓).
- On renvoie l'impôt à payer (arrondi à l'euro inf.), la décote et la réduction
  (arrondies à l'euro sup.).

Les nombres “magiques” sont regroupés en constantes, et chaque étape est isolée
dans des fonctions pour la clarté.
"""

from dataclasses import dataclass
from math import ceil, floor

#  ---------- Regulatory constants ----------

#  Professional expense allowance (floor/ceiling) for 2018 income
ABATTEMENT_TAUX = 0.10
ABATTEMENT_MIN = 437
ABATTEMENT_MAX = 12502

#  Scale 2018 (QF)
#  Each band is (borne_sup, rate). The first 0% bracket ends at €9,964.
BAREME_2018 = [
    (9964, 0.00),
    (27519, 0.14),
    (73779, 0.30),
    (156244, 0.41),
    (float("inf"), 0.45),
]

#  Family allowance ceiling
PF_QF_AVANTAGE_PAR_DEMI_PART = 1551  #  max benefit per 1/2 share ENFANT

#  Discount (thresholds according to number of adults / couple status)
DECOTE_SEUIL_CELIB = 1196
DECOTE_SEUIL_COUPLE = 1970

#  20% reduction: share-based thresholds
REDUCTION_BASE_1_PART = 18985
REDUCTION_BASE_2_PARTS = 37970
REDUCTION_PAR_DEMI_PART_SUP = 3803  #  addition of 1/2 share above the base of 1 or 2 shares


#  ---------- Data structures ----------

@dataclass
class ResultatImpot:
    impot: int  #  final tax payable, rounded down to the nearest euro
    decote: int  #  discount rounded up to the nearest euro
    reduction: int  #  20% discount rounded up to the nearest euro


#  ---------- Utility functions ----------

def arrondi_sup(euros: float) -> int:
    """Arrondi à l'euro supérieur (ceil)."""
    return int(ceil(euros))


def arrondi_inf(euros: float) -> int:
    """Arrondi à l'euro inférieur (floor)."""
    return int(floor(euros))


#  ---------- Step 1: Net taxable income (RNI) ----------

def calcul_abattement(revenus_bruts: float) -> int:
    """
    Calcule l'abattement de 10 %, plafonné/planche, ARRONDI À L'EURO SUPÉRIEUR.
    """
    abattement_calcule = revenus_bruts * ABATTEMENT_TAUX
    abattement_calcule = max(ABATTEMENT_MIN, min(ABATTEMENT_MAX, abattement_calcule))
    return arrondi_sup(abattement_calcule)


def revenu_net_imposable(revenus_bruts: float) -> int:
    """RNI = revenus bruts - abattement (abattement arrondi ↑)."""
    return int(round(revenus_bruts - calcul_abattement(revenus_bruts)))


#  ---------- Step 2: Number of units ----------

def nombre_parts(adultes: int, enfants: int) -> float:
    """
    Règles : 1 part si 1 adulte, 2 parts si 2 adultes.
    Enfants : +0,5 pour les 2 premiers, +1 par enfant à partir du 3e.
    (On ignore ici tout cas particulier type parent isolé, conformément au PDF.)
    """
    parts = 1.0 if adultes == 1 else 2.0
    if enfants <= 0:
        return parts
    if enfants == 1:
        return parts + 0.5
    #  children >= 2
    parts += 1.0  #  +0.5 +0.5 for the first two
    if enfants > 2:
        parts += (enfants - 2) * 1.0  #  +1 per child from the 3rd
    return parts


def demi_parts_enfants(enfants: int) -> int:
    """Nombre de DEMI-parts dû aux enfants (utile pour le plafonnement QF)."""
    if enfants <= 0:
        return 0
    if enfants == 1:
        return 1  #  0.5 part
    #  First two children = 2 * half-parts, then +2 half-parts per child from the 3rd onwards
    return 2 + (enfants - 2) * 2


#  ---------- Step 3: Gross tax via QF ----------

def impot_brut_par_parts(qf: float) -> float:
    """Impôt par part via barème progressif 2018 (sans arrondi)."""
    imp = 0.0
    previous = 0.0
    for borne_sup, taux in BAREME_2018:
        montant_tranche = max(0.0, min(qf, borne_sup) - previous)
        imp += montant_tranche * taux
        if qf <= borne_sup:
            break
        previous = borne_sup
    return imp


def impot_brut_total(rni: float, parts: float) -> float:
    """Impôt brut total = impôt par part * nombre de parts (sans arrondi)."""
    if parts <= 0:
        return 0.0
    qf = rni / parts
    return impot_brut_par_parts(qf) * parts


#  ---------- Step 4.1: Capping the family quotient ----------

def apply_plafonnement_qf(impot_avec_enfants: float, rni: float, adultes: int, enfants: int) -> float:
    """
    Calcule l'impôt après plafonnement QF si l'avantage des enfants dépasse le plafond.
    - Impôt A : avec parts incluant enfants
    - Impôt B : avec seulement parts des adultes (1 ou 2)
    - Avantage réel = B - A
    - Avantage maximal = 1551 € par 1/2 part ENFANT
    Si avantage réel > avantage max, impôt = B - avantage max, sinon impôt = A.
    """
    parts_adultes = 1.0 if adultes == 1 else 2.0
    imp_b = impot_brut_total(rni, parts_adultes)
    imp_a = impot_avec_enfants
    avantage_reel = max(0.0, imp_b - imp_a)
    avantage_max = PF_QF_AVANTAGE_PAR_DEMI_PART * demi_parts_enfants(enfants)
    if avantage_reel > avantage_max:
        return imp_b - avantage_max
    return imp_a


#  ---------- Step 4.2: Discount and 20% reduction ----------

def seuil_decote(adultes: int) -> int:
    return DECOTE_SEUIL_COUPLE if adultes >= 2 else DECOTE_SEUIL_CELIB


def seuil_reduction_20(rni: int, adultes: int, parts: float) -> int:
    """
    Seuil d'éligibilité à la réduction de 20 % :
    - Base : 18 985 € pour 1 part, 37 970 € pour 2 parts
    - + 3 803 € par 1/2 part supplémentaire au-delà de la base correspondante.
    """
    base = REDUCTION_BASE_2_PARTS if adultes >= 2 else REDUCTION_BASE_1_PART
    base_parts = 2.0 if adultes >= 2 else 1.0
    demi_sup = max(0, int(round((parts - base_parts) * 2)))
    return base + demi_sup * REDUCTION_PAR_DEMI_PART_SUP


def calc_decote_reduction(impot_apres_plaf: float, rni: int, adultes: int, parts: float):
    """
    Calcule (decote_arrondie, reduction_arrondie, impot_apres_correctifs).
    Règle d'interaction : la décote peut s'appliquer via repêchage si, après
    application THÉORIQUE de la réduction de 20 %, l'impôt passerait sous le seuil.
    Ordre : décote -> réduction.
    """
    imp = max(0.0, impot_apres_plaf)
    decote = 0
    reduction = 0

    #  Eligible for 20% discount based on RNI
    seuil_red = seuil_reduction_20(rni, adultes, parts)
    eligible_reduction = rni < seuil_red  #  "below a threshold" in PDF

    #  Eligible for discount: directly below threshold, or if reduction would cause the threshold to be lowered
    s_dec = seuil_decote(adultes)
    direct_decote = imp <= s_dec
    repechage = False
    if not direct_decote and eligible_reduction:
        imp_theorique_apres_red = imp * 0.80  #  theoretical 20% reduction
        if imp_theorique_apres_red <= s_dec:
            repechage = True

    elig_decote = (direct_decote or repechage) and (imp > 0)

    #  Apply discount first (if eligible)
    if elig_decote:
        montant_decote = s_dec - (imp * 0.75)
        decote_calc = max(0, arrondi_sup(montant_decote))
        #  The discount cannot exceed the remaining tax: it is limited to
        decote = min(decote_calc, arrondi_sup(imp))
        imp = max(0.0, imp - decote)

    #  Then apply the reduction if eligible
    if eligible_reduction and imp > 0:
        montant_reduction = imp * 0.20
        reduction = max(0, arrondi_sup(montant_reduction))
        imp = max(0.0, imp - reduction)

    return decote, reduction, imp


#  ---------- API main ----------

def calcule_impot(adultes: int, enfants: int, revenus_bruts: float) -> ResultatImpot:
    """
    Calcule l'impôt 2019 (revenus 2018) pour un foyer.
    Retourne ResultatImpot(impot, decote, reduction).
    """
    #  1) RNI
    rni = revenu_net_imposable(revenus_bruts)

    #  2) Units
    parts = nombre_parts(adultes, enfants)

    #  3) Gross tax via tax scale (with all shares)
    imp_brut_A = impot_brut_total(rni, parts)

    #  4.1) Family allowance ceiling
    imp_apres_plaf = apply_plafonnement_qf(imp_brut_A, rni, adultes, enfants)

    #  4.2) Discount then 20% reduction (with draft)
    decote, reduction, imp_corrige = calc_decote_reduction(imp_apres_plaf, rni, adultes, parts)

    #  5) Final tax rounding (↓) and bounding to 0
    imp_final = max(0, arrondi_inf(imp_corrige))

    return ResultatImpot(impot=imp_final, decote=decote, reduction=reduction)


#  ---------- Small test executable ----------

def _nearly_equal(a: int, b: int, tol: int = 1) -> bool:
    return abs(a - b) <= tol


def tests_unitaires():
    """
    Renvoie une liste de tuples (inputs, attendu, obtenu, ok) pour chaque test.
    Tolérance : ±1 € sur chaque valeur (impôt, décote, réduction).
    """
    cas = [
        #  (adults, children, income) -> (tax, discount, reduction)
        ((2, 2, 55555), (2815, 0, 0)),
        ((2, 2, 50000), (1385, 384, 346)),
        ((2, 3, 50000), (0, 720, 0)),
        ((1, 2, 100000), (19884, 0, 0)),
        ((1, 3, 100000), (16782, 0, 0)),
        ((2, 3, 100000), (9200, 0, 0)),
        ((2, 5, 100000), (4230, 0, 0)),
        ((1, 0, 100000), (22986, 0, 0)),
        ((2, 2, 30000), (0, 0, 0)),
        ((1, 0, 200000), (64211, 0, 0)),
        ((2, 3, 200000), (42843, 0, 0)),
        ((2, 2, 49500), (1297, 431, 324)),
    ]

    resultats = []
    for (adultes, enfants, revenus), attendu in cas:
        res = calcule_impot(adultes, enfants, revenus)
        obtenu = (res.impot, res.decote, res.reduction)
        ok = _nearly_equal(obtenu[0], attendu[0]) and _nearly_equal(obtenu[1], attendu[1]) and _nearly_equal(obtenu[2],
                                                                                                             attendu[2])
        resultats.append(((adultes, enfants, revenus), attendu, obtenu, ok))
    return resultats


if __name__ == "__main__":
    for inputs, attendu, obtenu, ok in tests_unitaires():
        print(f"{inputs} -> attendu={attendu}, obtenu={obtenu} : {'OK' if ok else 'ECHEC'}")

5.4. Problem 3

Now we ask ChatGPT to look up the tax calculation rules on the internet:

This time, we do not provide the PDF that contained the calculation rules to follow. We only provide our instructions in the text file. Note that this text file now contains 12 unit tests after adding, to the initial 11 tests, the one used by Gemini to demonstrate that my initial PDF was incorrect.

ChatGPT responds in 8 minutes, providing a link to download the generated script. Once loaded into PyCharm, this script passes all 12 tests. So for both problems posed, ChatGPT got the answers right on the first try, thereby outperforming Gemini.

ChatGPT provides its sources in its response:

There’s nothing more to say—it’s a job well done.

Now, we can ask it, just as we did with Gemini, to generate a PDF for students.

ChatGPT’s response came after several back-and-forth exchanges because the generated PDF used a font that replaced characters with squares. But eventually, it generated the PDF. I’m sharing it because it provides different rules from Gemini’s PDF, and I wondered which one was correct. Let’s investigate.

The difference from Gemini’s PDF lies in how the discount is calculated. The two AIs take different approaches. Gemini had written:

The two AIs have two different approaches. Which one is right?

5.5. Problem 4

We’ll ask ChatGPT to use its PDF to calculate the tax:

As in previous instances, it generates a Python script that works on the first try. We had added an additional test to the instructions:

1	`test13 : (1, 0, 18535) -> (359, 491, 90)`

All 13 tests were passed successfully.

5.6. Back to Gemini

Now, we return to Gemini, to which we will present ChatGPT’s PDF. Since the rules implemented in this PDF differ from those in Gemini’s PDF, we wonder what will happen:

Gemini first generated a Python script that failed some tests. We presented it with the logs:

Question 2

Question 3

There are still errors. Let’s continue.

Question 4

Still errors during execution:

This time it’s correct.

We’re still intrigued that, even with PDFs that have quite different calculation rules, both AIs generate correct results.

We ask Gemini the following question:

The full question is as follows:

Après tout ce qu'on a fait, compare les deux approches, la tienne et celle de ChatGPT dans les pDF que je te joins [.. selon Gemini] pour toi, [.. selon ChatGPT] pour ChatGPT. Dis-moi laquelle on doit préférer vu que sur les 13 tests générés tu les réussis tous avec l'un ou avec l'autre PDF.

Gemini’s answer is unequivocal:

5.7. What does ChatGPT think?

We ask ChatGPT the same question we asked Gemini.

ChatGPT’s response is as follows:

So, ChatGPT suggests a unit test to decide between the two methods. We duplicate:

The script [gemini3] generated by Gemini using its PDF [The Problem According to Gemini] as a source is duplicated in script [gemini4];
The [chatGPT3] script generated by ChatGPT using its PDF [The Problem According to ChatGPT] as a source is duplicated in the [chatGPT4] script;

Additionally, we add the unit test proposed by ChatGPT to each of the scripts [gemini4, chatGPT4] to distinguish between the two AIs.

Running [gemini4] yields the following results:


C:\Data\st-2025\dev\python\code\python-flask-2025-cours\.venv\Scripts\python.exe "C:/Program Files/JetBrains/PyCharm 2025.2.1.1/plugins/python-ce/helpers/pycharm/_jb_unittest_runner.py" --path "C:\Data\st-2025\dev\python\code\python-flask-2025-cours\outils ia\gemini\gemini4.py" 
Testing started at 17:45 ...
Launching unittests with arguments python -m unittest C:\Data\st-2025\dev\python\code\python-flask-2025-cours\outils ia\gemini\gemini4.py in C:\Data\st-2025\dev\python\code\python-flask-2025-cours
 
SubTest failure: Traceback (most recent call last):
  File "C:\Program Files\Python313\Lib\unittest\case.py", line 58, in testPartExecutor
    yield
  File "C:\Program Files\Python313\Lib\unittest\case.py", line 556, in subTest
    yield
  File "C:\Data\st-2025\dev\python\code\python-flask-2025-cours\outils ia\gemini\gemini4.py", line 234, in test_cas_verifies_simulateur_officiel
    self.assertAlmostEqual(calcul_impot, attendu_impot, delta=1, msg="Échec sur le montant de l'impôt")
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 2669 != 2270 within 1 delta (399 difference) : Échec sur le montant de l'impôt
 
 
 
 
Ran 1 test in 0.010s
 
FAILED (failures=1)
 
One or more subtests failed
Failed subtests list: [Test 'test12' avec entrée (2, 0, 43333)]
 
Process finished with exit code 1

So Gemini fails the test added by ChatGPT.

Running [chatGPT4] yields the following results:


C:\Data\st-2025\dev\python\code\python-flask-2025-cours\.venv\Scripts\python.exe "C:\Data\st-2025\dev\python\code\python-flask-2025-cours\outils ia\chatGPT\chatGPT4.py" 
Test (2, 2, 55555) -> obtenu (impôt=2814, décote=0, réduction=0) | attendu (2815, 0, 0) | OK
Test (2, 2, 50000) -> obtenu (impôt=1384, décote=384, réduction=347) | attendu (1385, 384, 346) | OK
Test (2, 3, 50000) -> obtenu (impôt=0, décote=721, réduction=0) | attendu (0, 720, 0) | OK
Test (1, 2, 100000) -> obtenu (impôt=19884, décote=0, réduction=0) | attendu (19884, 0, 0) | OK
Test (1, 3, 100000) -> obtenu (impôt=16782, décote=0, réduction=0) | attendu (16782, 0, 0) | OK
Test (2, 3, 100000) -> obtenu (impôt=9200, décote=0, réduction=0) | attendu (9200, 0, 0) | OK
Test (2, 5, 100000) -> obtenu (impôt=4230, décote=0, réduction=0) | attendu (4230, 0, 0) | OK
Test (1, 0, 100000) -> obtenu (impôt=22986, décote=0, réduction=0) | attendu (22986, 0, 0) | OK
Test (2, 2, 30000) -> obtenu (impôt=0, décote=0, réduction=0) | attendu (0, 0, 0) | OK
Test (1, 0, 200000) -> obtenu (impôt=64210, décote=0, réduction=0) | attendu (64211, 0, 0) | OK
Test (2, 3, 200000) -> obtenu (impôt=42842, décote=0, réduction=0) | attendu (42843, 0, 0) | OK
Test (2, 2, 49500) -> obtenu (impôt=1296, décote=431, réduction=325) | attendu (1297, 431, 324) | OK
Test (1, 0, 18535) -> obtenu (impôt=359, décote=491, réduction=90) | attendu (359, 491, 90) | OK
Test (2, 0, 43333) -> obtenu (impôt=2268, décote=0, réduction=401) | attendu (2270, 0, 400) | ECHEC
 Détails tolérance ±1€ : impôt ok? False, décote ok? True, réduction ok? True
 
Résultat global : AU MOINS UN TEST ÉCHOUE ❌
 
Process finished with exit code 0

ChatGPT also fails the added test, but not for the same reasons as Gemini. ChatGPT found the correct results but was off by 2 euros instead of the required 1 euro.

So from now on, we’ll use the PDF generated by ChatGPT with the following AIs. It’s worth noting that it’s because of the lack of unit tests in my instructions that both AIs passed the first tests. Hence, in this specific example, the importance of including unit tests for edge cases in tax calculation. Since it’s pretty hard to come up with these tests on your own. We’ll ask the AIs to add them themselves.

5.8. Problem 3 with unit tests generated by the AIs

The results obtained with Gemini and ChatGPT leave room for doubt. Did the AIs find a general solution that passes every conceivable test, or did they find a solution that only passes the required tests? We’ll start over with a solution without a PDF to force the AIs to go online and search for the information they need. And we’ll modify our instructions as follows:

The text file [instructionsSansPDF4.txt] already contains 14 required tests. To these tests, we add the following instructions:


7 - tu ajouteras autant de tests unitaires que nécessaires pour vérifier les cas limites du calcul de l'impôt.
 
Pour le code tu complèteras le script suivant auquel tu auras rajouté tes propres tests.
 
# =========================
# Tests unitaires (tolérance de ±1 €)
# =========================
 
TESTS = [
    # (adultes, enfants, revenus) -> (impot, decote, reduction)
    ((2, 2, 55555), (2815, 0, 0)),
    ((2, 2, 50000), (1385, 384, 346)),
    ((2, 3, 50000), (0, 720, 0)),
    ((1, 2, 100000), (19884, 0, 0)),
    ((1, 3, 100000), (16782, 0, 0)),
    ((2, 3, 100000), (9200, 0, 0)),
    ((2, 5, 100000), (4230, 0, 0)),
    ((1, 0, 100000), (22986, 0, 0)),
    ((2, 2, 30000), (0, 0, 0)),
    ((1, 0, 200000), (64211, 0, 0)),
    ((2, 3, 200000), (42843, 0, 0)),
    ((2, 2, 49500), (1297, 431, 324)),
    ((1, 0, 18535), (359, 491, 90)),
    ((2, 0, 43333), (2270, 0, 400)),
]
 
 
def _ok(a, b, tol=1):
    return abs(a - b) <= tol
 
 
def run_tests(verbose: bool = True) -> bool:
    all_ok = True
    for (params, expected) in TESTS:
        a, e, r = params
        exp_impot, exp_decote, exp_reduc = expected
        res = calcul_impot_2019(a, e, r)
        ok_impot = _ok(res.impot, exp_impot)
        ok_decote = _ok(res.decote, exp_decote)
        ok_reduc = _ok(res.reduction, exp_reduc)
        test_ok = ok_impot and ok_decote and ok_reduc
        if verbose:
            print(
                f"Test {params} -> obtenu (impôt={res.impot}, décote={res.decote}, réduction={res.reduction}) | attendu {expected} | {'OK' if test_ok else 'ECHEC'}")
            if not test_ok:
                print(
                    f" Détails tolérance ±1€ : impôt ok? {ok_impot}, décote ok? {ok_decote}, réduction ok? {ok_reduc}")
        all_ok &= test_ok
    if verbose:
        print("\nRésultat global :", "TOUS LES TESTS PASSENT ✅" if all_ok else "AU MOINS UN TEST ÉCHOUE ❌")
    return all_ok
 
 
if __name__ == "__main__":
    run_tests()

Lines 11–24: the 14 required tests;
Lines 5-55: this code comes from the script generated by ChatGPT. We will require Gemini to use this code to facilitate comparisons between the two generated scripts.

We’ll start with ChatGPT:

Its first response is incorrect. I tell it so by providing the execution logs:

Its second response is correct. ChatGPT added the following 11 tests to the 14 required tests:

# Cas limites supplémentaires (bords de paliers/arrondis)
TESTS += [
    # Abattement 10 % : plancher et plafond
    ((1, 0, 3000), (0, 0, 0)),  # 10 % = 300 < plancher 437 => RNI faible -> impôt nul
    ((1, 0, 200000), (64211, 0, 0)),  # plafond abattement déjà couvert dans tests initiaux

    # Décote : juste en dessous / au-dessus des seuils
    ((1, 0, 25000), None),  # diagnostique
    ((2, 0, 35000), None),  # diagnostique

    # Réduction 20 % : plein droit vs écrêtement
    ((1, 0, 17000), None),  # diagnostique
    ((2, 0, 34000), None),  # diagnostique
    ((1, 0, 20000), None),  # diagnostique
    ((2, 0, 40000), None),  # diagnostique

    # Changement de parts (plafonnement QF)
    ((2, 1, 80000), None),
    ((2, 2, 80000), None),
    ((2, 3, 80000), None),
]

There are now 25 unit tests. I manually verified the 11 new tests using the official DGIP simulator, and they pass.

Now, we’re moving on to Gemini. This is going to be much more complicated. It will manage to generate a script that passes all 25 ChatGPT tests, but only after a long debugging process.

Below is the debugging log:

Strangely, a majority of the tests failed, even among the 14 required ones, whereas in the past Gemini had generated code that passed them all.

The following response from Gemini is still incorrect:

Nor is the following response:

Nor is the following response. So I’m changing my approach. I’m asking it to pass the 25 tests that ChatGPT passed, attaching ChatGPT’s logs:

Gemini fails. It did add ChatGPT’s tests. I attach the logs of its execution:

Still no:

Still no, but it’s better:

Gemini is making new errors:

It’s improving again:

This time, it’s right:

Undoubtedly, in this specific example of calculating the 2019 tax with the constraints specified in the instruction file, ChatGPT was more accurate than Gemini. But this is just one example.

We can take it further. We can ask Gemini to regenerate a PDF based on the calculation rules it used to pass the 25 tests. We want to see if it has changed its initial reasoning regarding the calculations for the discount and the 20% reduction:

This time, Gemini generated a Markdown file that I then converted to PDF [The Problem According to Gemini Version 2]. And Gemini has indeed changed its reasoning:

We can see that the specific discount calculation and the carryover rule are no longer present. Gemini has now adopted ChatGPT’s reasoning.