What the agent does
evaluation_strategy was renamed eval_strategy in Transformers 4.41, kept as a deprecated alias with a FutureWarning that promised removal in 4.46, and then removed somewhat later: in a check, 4.50.3 still accepts the old name with the warning, while 4.53.3, 4.55.4 and the current 5.22.2 raise TypeError: TrainingArguments.__init__() got an unexpected keyword argument 'evaluation_strategy'. A training script with transformers>=4.28.0 in its requirements therefore works on one machine and fails on the next, at the moment TrainingArguments(...) is built, before a single batch is loaded.
FreedomIntelligence/GrammarGPT, a Chinese grammatical error correction project, is that script. requirements.txt lists transformers>=4.28.0, accelerate, datasets, fire, bitsandbytes, sentencepiece and peft straight from its Git repository, with no upper bound on any of them. finetune.py builds the trainer on lines 263 to 290: transformers.TrainingArguments(...) with per_device_train_batch_size, gradient_accumulation_steps, warmup_steps=4, num_train_epochs, learning_rate, fp16=False, logging_steps=5, optim="adamw_torch", evaluation_strategy="epoch" if val_set_size > 0 else "no" on line 276, save_strategy="epoch", save_total_limit=2, load_best_model_at_end, ddp_find_unused_parameters, group_by_length=group_by_length, report_to and run_name, wrapped in a transformers.Trainer with a DataCollatorForSeq2Seq. The function's defaults set val_set_size=500, so the branch that passes the old name is the one that runs.
The agent does not stop at the name in the error, because a TypeError reports only the first unknown keyword. It installs the requirements fresh, records the version that resolves, and then builds TrainingArguments once per keyword the script passes, each on its own, so every removed field shows up. On 5.22.2 that finds two: evaluation_strategy, and group_by_length, which the current release no longer has as a field at all. It also checks the fields that survive, optim, report_to, run_name, ddp_find_unused_parameters and save_total_limit among them, so the pull request can say the rest of the call is untouched for a reason.
Then it makes the smallest change that builds. evaluation_strategy becomes eval_strategy with the same conditional, and group_by_length is removed along with the function parameter that fed it, since the current Trainer has no sampler for it. The agent checks the one constraint that is easy to miss: load_best_model_at_end=True requires the save and eval strategies to match, and on the new release a missing eval_strategy defaults to no and raises ValueError at construction, so the two conditionals on val_set_size have to stay in step. It reads the installed Trainer signature, confirms that data_collator, train_dataset, eval_dataset and args are still the names used and that DataCollatorForSeq2Seq still takes tokenizer, pad_to_multiple_of, return_tensors and padding, sets a lower bound of the release it tested, and adds a test that builds the arguments on a temporary directory with val_set_size both zero and positive.
Start it with one API call
POST https://api.phoenix.so/o/{org}/sessions with this body:
{
"agent": "claude",
"input": "Our fine-tuning script fails as soon as it starts with TypeError: TrainingArguments.__init__() got an unexpected keyword argument 'evaluation_strategy'. Read the dependency files and the training script, and report which transformers, accelerate and torch versions a clean install resolves to. Reproduce the error in a clean virtual environment with a CPU build of torch, without loading any model or dataset: import the script's modules and build the TrainingArguments it builds. Because a TypeError names only the first unknown keyword, construct TrainingArguments once per keyword the script passes, each alone, and list every keyword the installed release rejects, with the release that removed it where the changelog says. Also read the installed Trainer and data collator signatures and list any parameter the script uses that has been renamed. Then make the smallest change that builds: rename evaluation_strategy to eval_strategy keeping the same condition, remove fields the installed release no longer has together with the script parameters that only fed them, and keep every other argument and value as it is. Check that load_best_model_at_end still has matching eval and save strategies in every branch. Set a lower bound on transformers to the release you tested. Add a test that builds the arguments in a temporary directory for both branches of the validation-set condition, run it, and include the output. Do not download models or datasets and do not start training. Open a pull request that lists each changed line, the versions installed, and what could not be verified without a GPU.",
"resources": [
{
"url": "https://github.com/your-org/your-app"
}
]
}- Get an API key. In the console, open Settings, then API keys, and create a key for your org. Send it in the Authorization header as a Bearer token.
- Send the request. POST the body in request.json to https://api.phoenix.so/o/{org}/sessions, with your repository as the resources URL. The run starts in its own cloud sandbox.
- Read what came back. Follow the run on its live stream or wait for the run.completed webhook, then read the result and its trajectory.
What comes back
A pull request that changes a few lines of the training script and the requirements file, plus one test. The description opens with the version a fresh install resolved to and the traceback from building the arguments, then a table of every keyword the script passes with one of three verdicts: accepted, renamed, or removed. That table is the point of the run, since the error message alone would have led to a one-word fix and a second failure on the next line.
A reviewer checks that the conditional on the validation set size is the same for eval_strategy and load_best_model_at_end, that nothing else in the TrainingArguments call changed value, and that removing group_by_length did not leave a command-line parameter that now does nothing without saying so. The test output shows both branches constructing.
The description ends with what a CPU sandbox cannot prove: that the model loads under the memory and quantisation settings the script uses, and that an epoch of training and evaluation completes on the GPU the project targets. Those are for the reviewer on the training machine.
Run it on every Monday morning
A training script has no CI and runs when someone needs it, often months apart, so a dependency with no upper bound breaks it silently in between. Give the repository's Project an automation with the prompt above and a schedule trigger with a weekly cron in your timezone. Each run installs the requirements fresh and builds the arguments and the trainer without a model; it opens a pull request only when a release has renamed or removed something, so most weeks cost one short run.
If a failed run on your training host is the first sign, forward its log from a handler on your side to a webhook trigger on the same automation with Authorization: Bearer and the trigger's secret. The log follows the saved prompt as event text, and the traceback in it tells the agent which keyword to start from.
The use case this job follows, with its automation and trigger: Fix a bug from an alert.
Why run it on Rush?
A one-keyword TypeError invites a one-keyword fix, and on a laptop with the old release installed nothing shows that the next keyword is gone too. The agent works in a sandbox with nothing installed, resolves the requirements the way a new machine would, and probes the constructor one argument at a time, so the pull request names every removed field at once instead of one per deploy.
It also keeps a GPU out of the loop until the end. Building TrainingArguments and reading the Trainer signature need only a CPU build of torch, so the whole check runs in minutes with its commands and output kept in the session, and the expensive step, a real training run, is left to the reviewer with a clear list of what remains to be seen.
Frequently asked questions
Is renaming evaluation_strategy to eval_strategy enough?
For that error, yes. But the installed release in the run also lacked group_by_length, which the same script passes, and a TypeError only reports the first unknown keyword. The agent probes every keyword so the second failure does not wait for the next run.
Which Transformers release removed evaluation_strategy?
The alias was deprecated in 4.41 with a warning naming 4.46. In the run, 4.50.3 still accepted it with the warning and 4.53.3 rejected it, so the removal landed between those two. Pinning below 4.53 restores the old name, at the cost of every later fix.
Why did the fix then fail with a ValueError about load_best_model_at_end?
Because load_best_model_at_end=True requires the eval and save strategies to match, and when eval_strategy is left out it defaults to no while save_strategy is epoch. The agent keeps both conditionals on the same variable so they move together.
Does the agent run the training?
No. It builds the arguments and the trainer without a model or dataset and runs the new test on a CPU build of torch. Loading the base model, quantisation and the epoch itself need the GPU machine and are listed for the reviewer.
Will it change other training settings while it is there?
No. Every value that the installed release still accepts stays as it is, and the pull request lists the accepted keywords so that is visible. Changing hyperparameters is a separate request.
More jobs like this: Fix a bug from an alert
- Fix "expected exactly 5 fields, found 6" After Upgrading robfig/cron to v3 (robfig/cron)
- Fix "error: externally-managed-environment" in a Docker Build (Docker)
- Fix the Manifest Behind a Failed Argo CD Sync (Argo CD)
- Fix a Failing CI Build in an Angular App (Angular)
- Fix a Failing Molecule CI Run for an Ansible Role (Ansible)
- Fix Failing GitHub Actions Checks in an ASP.NET Core App (ASP.NET Core)