# Fix "TrainingArguments.__init__() got an unexpected keyword argument 'evaluation_strategy'"

Yes. An agent renames evaluation_strategy to eval_strategy, finds the other TrainingArguments fields the new release dropped, and proves the call builds before a PR.

[Run this on Rush](https://cloud.getrush.ai/run/fix-evaluation-strategy-unexpected-keyword-argument/transformers/). Opens the console with this prompt ready to send, in a Project that has your repository.

## What the agent does

`evaluation_strategy` was renamed `eval_strategy` in Transformers 4.41, kept as a deprecated alias with a `FutureWarning` that promised removal in 4.46, and then removed somewhat later: in a check, 4.50.3 still accepts the old name with the warning, while 4.53.3, 4.55.4 and the current 5.22.2 raise `TypeError: TrainingArguments.__init__() got an unexpected keyword argument 'evaluation_strategy'`. A training script with `transformers>=4.28.0` in its requirements therefore works on one machine and fails on the next, at the moment `TrainingArguments(...)` is built, before a single batch is loaded.

FreedomIntelligence/GrammarGPT, a Chinese grammatical error correction project, is that script. `requirements.txt` lists `transformers>=4.28.0`, `accelerate`, `datasets`, `fire`, `bitsandbytes`, `sentencepiece` and `peft` straight from its Git repository, with no upper bound on any of them. `finetune.py` builds the trainer on lines 263 to 290: `transformers.TrainingArguments(...)` with `per_device_train_batch_size`, `gradient_accumulation_steps`, `warmup_steps=4`, `num_train_epochs`, `learning_rate`, `fp16=False`, `logging_steps=5`, `optim="adamw_torch"`, `evaluation_strategy="epoch" if val_set_size > 0 else "no"` on line 276, `save_strategy="epoch"`, `save_total_limit=2`, `load_best_model_at_end`, `ddp_find_unused_parameters`, `group_by_length=group_by_length`, `report_to` and `run_name`, wrapped in a `transformers.Trainer` with a `DataCollatorForSeq2Seq`. The function's defaults set `val_set_size=500`, so the branch that passes the old name is the one that runs.

The agent does not stop at the name in the error, because a `TypeError` reports only the first unknown keyword. It installs the requirements fresh, records the version that resolves, and then builds `TrainingArguments` once per keyword the script passes, each on its own, so every removed field shows up. On 5.22.2 that finds two: `evaluation_strategy`, and `group_by_length`, which the current release no longer has as a field at all. It also checks the fields that survive, `optim`, `report_to`, `run_name`, `ddp_find_unused_parameters` and `save_total_limit` among them, so the pull request can say the rest of the call is untouched for a reason.

Then it makes the smallest change that builds. `evaluation_strategy` becomes `eval_strategy` with the same conditional, and `group_by_length` is removed along with the function parameter that fed it, since the current `Trainer` has no sampler for it. The agent checks the one constraint that is easy to miss: `load_best_model_at_end=True` requires the save and eval strategies to match, and on the new release a missing `eval_strategy` defaults to `no` and raises `ValueError` at construction, so the two conditionals on `val_set_size` have to stay in step. It reads the installed `Trainer` signature, confirms that `data_collator`, `train_dataset`, `eval_dataset` and `args` are still the names used and that `DataCollatorForSeq2Seq` still takes `tokenizer`, `pad_to_multiple_of`, `return_tensors` and `padding`, sets a lower bound of the release it tested, and adds a test that builds the arguments on a temporary directory with `val_set_size` both zero and positive.

## Start it with one API call

`POST https://api.phoenix.so/o/{org}/sessions`

```json
{
  "agent": "claude",
  "input": "Our fine-tuning script fails as soon as it starts with TypeError: TrainingArguments.__init__() got an unexpected keyword argument 'evaluation_strategy'. Read the dependency files and the training script, and report which transformers, accelerate and torch versions a clean install resolves to. Reproduce the error in a clean virtual environment with a CPU build of torch, without loading any model or dataset: import the script's modules and build the TrainingArguments it builds. Because a TypeError names only the first unknown keyword, construct TrainingArguments once per keyword the script passes, each alone, and list every keyword the installed release rejects, with the release that removed it where the changelog says. Also read the installed Trainer and data collator signatures and list any parameter the script uses that has been renamed. Then make the smallest change that builds: rename evaluation_strategy to eval_strategy keeping the same condition, remove fields the installed release no longer has together with the script parameters that only fed them, and keep every other argument and value as it is. Check that load_best_model_at_end still has matching eval and save strategies in every branch. Set a lower bound on transformers to the release you tested. Add a test that builds the arguments in a temporary directory for both branches of the validation-set condition, run it, and include the output. Do not download models or datasets and do not start training. Open a pull request that lists each changed line, the versions installed, and what could not be verified without a GPU.",
  "resources": [
    {
      "url": "https://github.com/your-org/your-app"
    }
  ]
}
```

1. **Get an API key.** In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
2. **Send the request.** POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
3. **Read what came back.** Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.

## What comes back

A pull request that changes a few lines of the training script and the requirements file, plus one test. The description opens with the version a fresh install resolved to and the traceback from building the arguments, then a table of every keyword the script passes with one of three verdicts: accepted, renamed, or removed. That table is the point of the run, since the error message alone would have led to a one-word fix and a second failure on the next line.

A reviewer checks that the conditional on the validation set size is the same for `eval_strategy` and `load_best_model_at_end`, that nothing else in the `TrainingArguments` call changed value, and that removing `group_by_length` did not leave a command-line parameter that now does nothing without saying so. The test output shows both branches constructing.

The description ends with what a CPU sandbox cannot prove: that the model loads under the memory and quantisation settings the script uses, and that an epoch of training and evaluation completes on the GPU the project targets. Those are for the reviewer on the training machine.

## Run it on every Monday morning

A training script has no CI and runs when someone needs it, often months apart, so a dependency with no upper bound breaks it silently in between. Give the repository's Project an automation with the prompt above and a `schedule` trigger with a weekly `cron` in your `timezone`. Each run installs the requirements fresh and builds the arguments and the trainer without a model; it opens a pull request only when a release has renamed or removed something, so most weeks cost one short run.

If a failed run on your training host is the first sign, forward its log from a handler on your side to a `webhook` trigger on the same automation with `Authorization: Bearer` and the trigger's secret. The log follows the saved prompt as event text, and the traceback in it tells the agent which keyword to start from.

## Why run it on Rush?

A one-keyword `TypeError` invites a one-keyword fix, and on a laptop with the old release installed nothing shows that the next keyword is gone too. The agent works in a sandbox with nothing installed, resolves the requirements the way a new machine would, and probes the constructor one argument at a time, so the pull request names every removed field at once instead of one per deploy.

It also keeps a GPU out of the loop until the end. Building `TrainingArguments` and reading the `Trainer` signature need only a CPU build of torch, so the whole check runs in minutes with its commands and output kept in the session, and the expensive step, a real training run, is left to the reviewer with a clear list of what remains to be seen.

## Frequently asked questions

### Is renaming evaluation_strategy to eval_strategy enough?

For that error, yes. But the installed release in the run also lacked `group_by_length`, which the same script passes, and a `TypeError` only reports the first unknown keyword. The agent probes every keyword so the second failure does not wait for the next run.

### Which Transformers release removed evaluation_strategy?

The alias was deprecated in 4.41 with a warning naming 4.46. In the run, 4.50.3 still accepted it with the warning and 4.53.3 rejected it, so the removal landed between those two. Pinning below 4.53 restores the old name, at the cost of every later fix.

### Why did the fix then fail with a ValueError about load_best_model_at_end?

Because `load_best_model_at_end=True` requires the eval and save strategies to match, and when `eval_strategy` is left out it defaults to `no` while `save_strategy` is `epoch`. The agent keeps both conditionals on the same variable so they move together.

### Does the agent run the training?

No. It builds the arguments and the trainer without a model or dataset and runs the new test on a CPU build of torch. Loading the base model, quantisation and the epoch itself need the GPU machine and are listed for the reviewer.

### Will it change other training settings while it is there?

No. Every value that the installed release still accepts stays as it is, and the pull request lists the accepted keywords so that is visible. Changing hyperparameters is a separate request.

## More jobs like this: Fix a bug from an alert

- [Fix "expected exactly 5 fields, found 6" After Upgrading robfig/cron to v3](https://getrush.ai/run/fix-expected-exactly-5-fields-found-6/robfig-cron/)
- [Fix "error: externally-managed-environment" in a Docker Build](https://getrush.ai/run/fix-externally-managed-environment/docker/)
- [Fix the Manifest Behind a Failed Argo CD Sync](https://getrush.ai/run/fix-failed-syncs/argo-cd/)
- [Fix a Failing CI Build in an Angular App](https://getrush.ai/run/fix-failing-ci/angular/)
- [Fix a Failing Molecule CI Run for an Ansible Role](https://getrush.ai/run/fix-failing-ci/ansible/)
- [Fix Failing GitHub Actions Checks in an ASP.NET Core App](https://getrush.ai/run/fix-failing-ci/aspnet-core/)

## Keep reading

- [How a Rush run works on your repository, from request to pull request](https://getrush.ai/cloud/)
- [Compare cloud agents, agent sandboxes and AI code review](https://getrush.ai/compare/)
- [Rush pricing: Free, Pro and Max](https://getrush.ai/pricing/)
