LoRA is often introduced as a way to fine-tune a large model without updating all its weights. That description is correct, but it leaves several practical questions unanswered: what is actually trained, how much memory is saved, what does the adapter contain, and how do we tell whether training improved the task?
This article connects the mathematics to a small support-ticket experiment. I used a local checkpoint identified in my scripts as Qwen3.5-9B, Transformers, PEFT, and TRL. The saved training script uses a four-bit quantized base with LoRA, so the experiment is more precisely described as QLoRA.
The surviving terminal log records two epochs and twelve optimizer updates. It shows falling training and validation losses. It does not establish a production classification accuracy.
One detail matters: the available data excerpt contains Chinese category names, while a later inference script asks for internal error codes. Those are different output contracts. I preserve that distinction rather than presenting them as one verified end-to-end result.
1. What LoRA Changes in a Linear Layer
For a linear layer, ignoring bias, use the column-vector convention:
with:
A matrix with two rows and three columns maps a three-dimensional input to a two-dimensional output. The multiplication is (Wx), not (xW), under this convention.
Full fine-tuning updates the model’s original weights. LoRA freezes selected base weights and represents an additive update through two smaller trainable matrices:
For standard LoRA, define:
Then:
The rank of (BA) is at most (r); it does not have to equal (r).
Figure 1. The adapter branch produces a vector (sBAx). Its weight update is the matrix (sBA). Bias and dropout are omitted.
There is no separate compartment containing “old knowledge” and another containing “new knowledge.” The two paths describe arithmetic. Enabling an adapter changes behavior and can also degrade behavior that was useful before training.
How many parameters does the update save?
Suppose a selected weight matrix has shape (4096\times4096). Updating it directly involves:
parameters. With (r=16), the two LoRA matrices contain:
parameters, or:
of that matrix’s parameter count.
This is a per-layer parameter comparison, not a claim that total GPU memory drops to 0.78125%. Base weights, activations, temporary buffers, and other allocations still consume memory.
The low-rank update is a useful restriction, not a universal law about every task. Effectiveness depends on the base, data, target layers, and required adaptation. The original LoRA paper provides the method and empirical motivation.
2. Initialization: Why One Matrix Starts at Zero
A common initialization is to initialize (A) randomly and (B) to zero. Initially:
The added branch contributes zero. Under otherwise identical inference conditions, attaching this initially zero adapter does not change the layer’s output.
Why not initialize both matrices to zero? Let (G) denote the loss gradient with respect to (\Delta W). For the standard factorization:
If both factors are zero, both task-loss gradients are zero. With random (A) and zero (B), the gradient for (B) can be nonzero, allowing learning to begin. The task-loss gradient for (A) can become nonzero after (B) changes.
This is the conventional initialization, not the only option. PEFT supports other initialization methods; see the PEFT LoRA reference.
3. LoRA, QLoRA, and SFT Describe Different Choices
| Term | What it describes |
|---|---|
| SFT | Supervised fine-tuning on examples of desired outputs |
| LoRA | A parameter-efficient way to represent and train weight updates |
| QLoRA | Training low-rank adapters through a frozen quantized base |
| Full fine-tuning | Updating the original model parameters |
| DPO | A preference-training objective that can also use parameter-efficient updates |
My experiment used conversational supervised examples and low-rank adapters over a four-bit base: SFT using QLoRA.
QLoRA reduces base-weight storage compared with loading the base entirely in a higher precision. It does not mean all arithmetic happens in four bits. This script uses NF4 quantization, double quantization, and BF16 computation.
Freezing base parameters prevents their updates. It does not remove the base from the computation or eliminate the need to propagate gradients through relevant operations. The QLoRA paper explains the quantized-base approach.
4. The Configuration Choices That Matter
Rank and alpha
I used (r=16) and , giving a standard LoRA scale of .
Larger rank increases available update capacity and parameter count, but does not guarantee better validation performance. Alpha scales the branch; the learned matrices, optimizer, and learning rate also influence the effective update.
Ranks such as 8, 16, and 32 can form a small comparison set for a new experiment, rather than a universal recommendation. The optional rsLoRA variant uses ; it was not enabled here.
Target modules
The script requests these module names:
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
]Names are architecture-dependent. A configuration list is not proof that every intended layer was adapted, especially in architectures combining different attention or sequence-processing components.
For a rerun, inspect the matched modules and print the trainable parameter count. Do not assume a list copied from another model gives complete coverage.
Dropout, learning rate, and sequence length
The recorded settings were dropout 0.05, learning rate ), two epochs, and maximum sequence length 256.
A shorter sequence limit can reduce work for short tickets, but it applies to the formatted conversation, not just the user’s message. Inspect tokenized examples to ensure truncation does not remove the answer.
More epochs and lower training loss are not automatically better. The important result is whether unseen tickets receive the correct label in the required format.
5. Define a Stable Output Contract
The available data excerpt uses five Chinese labels:
| Stored label | English explanation |
|---|---|
| 支付异常 | Payment issue |
| 退款异常 | Refund issue |
| 物流异常 | Shipping issue |
| 账号异常 | Account issue |
| 商品异常 | Product issue |
The labels remain in Chinese because they are part of the actual dataset. Translating the article should not silently change its training targets.
An example record is:
{"messages":[{"role":"system","content":"你是工单分类助手，只输出分类名称。"},{"role":"user","content":"付款成功以后订单还是待支付"},{"role":"assistant","content":"支付异常"}]}The system instruction means “You are a ticket classification assistant. Output only the category name.” The user says that payment succeeded but the order still appears unpaid.
The available training excerpt contains ten records, two per category. The validation excerpt contains one payment-related record. These are excerpts, not the complete data that produced the screenshot.
What about internal codes such as PAY_101?
Fine-grained internal codes are a reasonable extension. A business might distinguish “charged but order unpaid” from “duplicate charge.”
However, a later saved inference script requests an internal code, whereas the surviving excerpt teaches category names. I cannot establish that the screenshot came from a complete code-labeled dataset.
The category task is therefore the concrete data example here. A code-classification experiment would need its own documented codebook, matching training targets, and evaluation labels. It should not inherit the category experiment’s metrics without evidence.
Preparing a complete dataset
Use a layout such as:
data/
├── train.jsonl
├── validation.jsonl
└── test.jsonlThe historical script uses the first two files. The third is recommended for a later independent evaluation.
Split related examples by source ticket or paraphrase family, not merely by row. Otherwise, nearly identical rewrites may appear in training and evaluation.
Include ambiguous and out-of-scope tickets. Decide whether they require clarification, an explicit fallback label, or human review. The five-label excerpt does not establish such a policy.
6. Establish a Useful Baseline
Ollama was part of the original local-inference workflow. Transformers and PEFT were used for training with a local Hugging Face-style checkpoint.
Those environments can differ in quantization, templates, and generation settings. Comparing “Ollama before” with “Transformers after” would mix several variables.
A stronger follow-up comparison is:
| Condition | Purpose |
|---|---|
| Base model with category definitions and a few examples | A useful prompting baseline |
| Base plus adapter, using a documented prompt | Measure the adapted system |
| Final deployment artifact | Check conversion and quantization effects |
The first two should use the same base revision, runtime, precision, and decoding settings when possible. For a strict adapter-effect comparison, hold the prompt constant too. If the adapted system uses a shorter prompt, report that separately as a system-level trade-off.
Asking the base to invent undisclosed internal codes does not test whether prompting could solve the task. Supply the output contract.
I did not recover a complete before-and-after prediction table, so this article reports training observations rather than an invented accuracy improvement.
7. The Recorded QLoRA Training Setup
The surviving script points to a local directory:
models/Qwen3.5-9BThat identifies the intended checkpoint in my workflow. A directory name does not independently establish its exact revision or configuration. Historical package versions and hardware details were not preserved with these materials.
The current Qwen3.5-9B model card describes a multimodal model. For a new installation, verify the model class and processor/tokenizer path against the checkpoint and installed libraries. The historical script used AutoModelForCausalLM and AutoTokenizer; it is a record of that setup, not a compatibility promise for every library version.
| Setting | Recorded value |
|---|---|
| Base loading | Four-bit NF4 |
| Double quantization | Enabled |
| Compute dtype | BF16 |
| Rank / alpha / dropout | 16 / 32 / 0.05 |
| Per-device training batch | 1 |
| Gradient accumulation | 16 |
| Learning rate | 1e-4 |
| Maximum length | 256 |
| Epochs | 2 |
| Logging | Every optimizer step |
| Evaluation | Every 2 steps |
| Checkpoint saving | Every 4 steps |
| Gradient checkpointing | Enabled |
On one training device, a full accumulation group represents sixteen examples. Final partial groups can differ.
Load and prepare the quantized base
import torch
from datasets import load_dataset
from peft import LoraConfig, prepare_model_for_kbit_training
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
BitsAndBytesConfig,
)
from trl import SFTConfig, SFTTrainer
model_path = "models/Qwen3.5-9B"
tokenizer = AutoTokenizer.from_pretrained(
model_path,
local_files_only=True,
)
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
model_path,
quantization_config=bnb_config,
device_map="auto",
local_files_only=True,
torch_dtype=torch.bfloat16,
)
model = prepare_model_for_kbit_training(
model,
use_gradient_checkpointing=True,
)The log includes a deprecation warning about torch_dtype, consistent with the historical script. A maintained version should use the argument supported by its pinned Transformers release.
BF16 requires suitable hardware support. Changing the Trainer to FP16 alone is not a complete precision change: the quantization compute dtype and other settings must also be reviewed.
The automatic device mapping is reproduced from the script. Do not interpret it as a general multi-GPU training recipe; distributed training requires a supported placement strategy.
Load data and attach LoRA
dataset = load_dataset(
"json",
data_files={
"train": "data/train.jsonl",
"validation": "data/validation.jsonl",
},
)
lora_config = LoraConfig(
task_type="CAUSAL_LM",
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
)Train and save
train_config = SFTConfig(
output_dir="outputs/qwen35-ticket-qlora",
num_train_epochs=2,
per_device_train_batch_size=1,
gradient_accumulation_steps=16,
learning_rate=1e-4,
max_length=256,
logging_steps=1,
eval_strategy="steps",
eval_steps=2,
save_steps=4,
save_total_limit=2,
gradient_checkpointing=True,
bf16=True,
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=train_config,
train_dataset=dataset["train"],
eval_dataset=dataset["validation"],
processing_class=tokenizer,
peft_config=lora_config,
)
trainer.train()
trainer.save_model()These blocks follow the saved training configuration. The complete companion script is included with this article. With the compatible environment, checkpoint, and complete dataset in place, run it from a project root containing the models and data directories:
python code/train_qlora.pyThe output directory is outputs/qwen35-ticket-qlora. Standard PEFT artifacts include adapter_config.json and adapter_model.safetensors. Checkpoints written during training are separate from the final save.
The supplied data-excerpt directory is deliberately not named data: copying its ten training rows into a run will not reproduce the historical twelve-step experiment.
For a rerun, record Python, PyTorch, Transformers, PEFT, TRL, Accelerate, and bitsandbytes versions, hardware, checkpoint revision, dataset hashes, matched adapter modules, and trainable parameter count.
Also verify the loss mask. The historical script does not explicitly request assistant-only loss. Whether prompt and system tokens contribute depends on the relevant TRL behavior and formatting configuration. Conversational data alone does not prove that loss was computed only on the answer. See the SFTTrainer documentation.
8. What the Training Screenshot Shows
Figure 2. Original terminal capture. The figures below are transcribed from the log; this is not a newly generated benchmark.
| Observation | Approximate recorded value |
|---|---|
| First logged training loss | 3.024 |
| Last logged training loss | 0.941 |
| First logged validation loss | 2.427 |
| Last logged validation loss | 1.143 |
| Validation token accuracy | 53% to 78% |
| Completed training | 12 optimizer steps, 2 epochs |
The first validation result occurs after training has already started. It is not an untouched-base baseline. The first and last training-loss values are logging observations, not full-epoch averages.
How large was the dataset?
My earlier notes describe approximately 87 training examples and 25 validation examples. The screenshot is consistent with that:
The first logged epoch fraction is approximately 0.1839. With batch size one and sixteen accumulation steps on one device, that is consistent with 87 examples. Two epochs with roughly six optimizer updates each are consistent with the twelve-step run.
Validation timing and throughput also suggest approximately 25 examples.
This is a reconstruction from logs and notes, not a fresh count of the complete dataset. The downloadable excerpts contain only ten training examples and one validation example.
What can we conclude?
Training and validation losses decreased over the observed run. There was no sustained rise in recorded validation loss while training loss fell.
That supports the limited conclusion that optimization progressed on these data. It does not establish:
- A 78% ticket-classification accuracy.
- Absence of overfitting.
- Reliable performance on new business traffic.
- An advantage over a well-designed prompt.
- Successful learning of the internal-code scheme.
Token accuracy concerns predicted tokens under the evaluation procedure. It is not the same as generating a correct complete category or code.
With about 25 validation examples, even a generated-label metric would be coarse: one example changes accuracy by four percentage points.
9. Load the Adapter and Inspect a Prediction
The saved inference script loads the base in BF16, attaches the adapter, sets evaluation mode, disables sampling, and allows up to 512 generated tokens.
That differs from the four-bit base used during training. It is a configuration to investigate, not an identical numerical setting. Evaluate the precision that will actually be deployed.
For the category-name data shown here, the following revised smoke test uses the matching output instruction. It improves the original code-requesting test; it is not a recovered historical prediction.
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_path = "models/Qwen3.5-9B"
adapter_path = "outputs/qwen35-ticket-qlora"
tokenizer = AutoTokenizer.from_pretrained(
base_path, local_files_only=True
)
base_model = AutoModelForCausalLM.from_pretrained(
base_path,
dtype=torch.bfloat16,
device_map="auto",
local_files_only=True,
)
model = PeftModel.from_pretrained(base_model, adapter_path)
model.eval()
messages = [
{
"role": "system",
"content": "你是工单分类助手，只输出分类名称。",
},
{
"role": "user",
"content": "银行卡已经扣款，订单状态一直没有更新为已支付",
},
]
inputs = tokenizer.apply_chat_template(
messages,
tokenize=True,
add_generation_prompt=True,
enable_thinking=False,
return_tensors="pt",
return_dict=True,
)
inputs = {
key: value.to(model.device)
for key, value in inputs.items()
}
with torch.inference_mode():
output = model.generate(
**inputs,
max_new_tokens=512,
do_sample=False,
repetition_penalty=1.05,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
new_tokens = output[0][inputs["input_ids"].shape[1]:]
answer = tokenizer.decode(
new_tokens, skip_special_tokens=True
).strip()
print("Model output:", answer)
print("Expected label:", "支付异常")
print("Exact match:", answer == "支付异常")
print("Generated tokens:", len(new_tokens))The expected label comes from the available validation example; it is not a claim that this revised script produced that output. This example is also not an independent held-out test.
Confirm that the checkpoint’s chat template supports and honors the thinking setting. Accepting a keyword without raising an error does not prove that thinking was disabled.
A 512-token ceiling avoids the earlier very short generation limit, but it is not the desired length of a label. Record truncation and extra text rather than silently extracting a convenient label from a long answer.
10. Turn the Smoke Test into a Task Evaluation
A single prediction checks loading and generation. A classifier needs a test set and explicit success criteria.
For (N) examples, exact-label accuracy is:
Specify normalization before testing. Trimming surrounding whitespace is reasonable; removing explanations or guessing an intended label can hide output-contract failures.
Report at least:
| Metric | What it answers |
|---|---|
| Exact-label accuracy | Was the complete label correct? |
| Valid-label rate | Was the output one allowed label and nothing else? |
| Per-class correct / total | Which categories fail? |
| Confusion matrix | Which classes are being mixed up? |
| Ambiguous and out-of-scope handling | Does the system follow its fallback policy? |
Keep row-level predictions with the input, expected label, raw output, prompt version, model condition, and correctness.
The original inference question about a rejected refund illustrates why the task definition matters. It may fit a broad “refund issue” category, but it is not automatically one of a finer codebook’s specific refund statuses. Broad-category and fine-code experiments require different gold labels.
This full evaluation is a proposed next step. No generated-label scores from it are claimed here.
11. Keep the Adapter Separate or Merge It?
A PEFT adapter typically includes a configuration file and an adapter weight file. It still requires a compatible base checkpoint. Additional saved modules can make the artifact larger than the low-rank factors alone.
Figure 3. A and B are factors within an adapter, not separate interchangeable adapters. Here ; use the trained configuration’s scale.
Keeping the adapter separate is useful for versioning, rollback, and selecting compatible task adapters. Runtime support determines whether and how switching works.
For a standard mergeable LoRA layer:
Inference can then use the merged weight without that separate LoRA branch.
With a quantized training base, dequantization, merging into a higher-precision checkpoint, conversion, and requantization can change numerical behavior. Ideal algebraic equivalence is not a guarantee of identical outputs across deployment artifacts.
Ollama can be a final inference target when the architecture and converted format are supported. Record the base revision, merge method, conversion tool, quantization, template, and runtime version, then repeat evaluation. Consult the Ollama import documentation for the supported import path.
The retained materials do not establish a successful merged-model conversion and Ollama deployment. That remains a stage to verify, not a completed result.
12. When Would I Choose LoRA for This Task?
For a small, stable set of categories, I would first test clear instructions, a few examples, and potentially a smaller classifier. A large model plus an adapter may not be necessary.
LoRA becomes more interesting when there is representative data, a stable output contract, and an identified limitation that simpler baselines do not address. It may help with formatting or task-specific distinctions, but those benefits need measurement.
If rules change frequently, keeping them in prompts, retrieval, or application logic may be easier to maintain than retraining an adapter. Fine-tuning should not hide an unclear category definition.
This experiment demonstrates the mechanics of a small QLoRA run and preserves evidence that optimization progressed. The next milestone is a controlled comparison showing that the adapted system handles unseen tickets more reliably under a consistent task definition.
Related reading: Text embeddings and retrieval quality | Understanding Transformers and self-attention