Tapirus Open 4B: fine tuning and RAG on Qwen3-4B for a domain general LLMs don't cover

General-purpose models answer almost anything — which is exactly why they answer poorly about refaunation. When the question involves species reintroduction, seed dispersal by frugivores, or the restoration of ecological interactions, what comes out is usually fluent, plausible and inaccurate. That friction is what led me to train a model of my own.

The result is Tapirus Open 4B: an open-weights small language model specialized in refaunation, biodiversity conservation and the restoration of ecological interactions. It is built on Qwen/Qwen3-4B-Instruct-2507, which I fine-tuned and then paired with a RAG layer. This post is about the engineering decisions behind it — and what I would do differently.

The blind spot of general-purpose models

An LLM trained mostly on the open web learns the distribution of what is abundant. Technical literature on restoration ecology is the opposite: low volume, its own vocabulary, and a lot of regional nuance. Terms like defaunation, dispersal syndrome, ecological baseline or extinct megafauna show up rarely — and when they do, they arrive detached from any applied context.

In practice that produces two kinds of error. The first is obvious: the model invents interactions between species that don’t exist. The second is more dangerous because it slips through: the model gets the general concept right but the scale, region or species wrong — and well-written text carrying a wrong fact is worse than an honest refusal.

There were two paths: tune prompts on top of a large model, or build something specialized. Prompt engineering improves the shape of an answer, but it cannot create knowledge that isn’t there.

Why an SLM instead of a large LLM

• In a narrow scope, specialization beats size.The model doesn’t need to write SQL or discuss foreign policy. It needs to get one domain right.
• Cost and latency. A 4B model is fast and cheap enough to actually be used, not just demoed.
• Local execution.It runs on a modest GPU and, when quantized, even on a laptop. For field research — limited connectivity, data that can’t always leave the environment — that stops being a detail and becomes a requirement.
• Reproducibility. Open weights let someone else verify, challenge and continue the work.

Why Qwen3-4B-Instruct-2507

Picking the base model had less to do with benchmark rankings than with the project’s concrete constraints:

• 4 billion parameters is the sweet spot between reasoning capability and affordable hardware.
• The Instruct variant already arrives aligned for dialogue, so fine tuning starts from useful conversational behavior instead of building it from scratch.
• Consistent performance in Portuguese, the language of both the corpus and the Refauna audience.
• A long context window, which leaves room for retrieved passages without squeezing out the user’s question.
• A permissive license, a precondition for publishing a derivative with open weights.

Choosing a base model is an architecture decision, not a scoreboard one. License, language, hardware and what you intend to publish afterwards weigh more than two points on a benchmark.

Fine tuning and RAG don’t compete: they solve different problems

This may be the most common confusion in applied AI projects. The two techniques get framed as alternatives — “should I fine-tune or use RAG?” — when they actually address different layers of the problem.

• Fine tuning changes how the model speaks. Tone, answer structure, technical vocabulary, and — most underrated — its willingness to refuse what falls outside its scope.
• RAG changes what the model knows right now. Verifiable facts, with a citable source, updatable without retraining anything.

The rule of thumb I use: if the correct answer changes when a new document enters the corpus, it’s a RAG problem. If the model has the information but expresses it badly, outside the domain’s vocabulary, it’s a fine tuning problem. Refauna had both — hence both techniques.

The pipeline, in four stages

1. Data preparation

The Refauna corpus was converted into instruction/response pairs using the base model’s own chat template. Format consistency matters more than volume here: heterogeneous examples teach the model to be heterogeneous.

{"messages": [
  {"role": "system", "content": "Você é um assistente especializado em refaunação e restauração de interações ecológicas."},
  {"role": "user", "content": "Por que a perda de frugívoros de grande porte compromete a regeneração da floresta?"},
  {"role": "assistant", "content": "..."}
]}

2. Supervised fine tuning

Training runs on top of the Instruct variant, preserving the conversational behavior it already has. The main risk at this stage is catastrophic forgetting: overtraining on a narrow domain degrades general abilities you still want, like following instructions and writing well. Short runs with frequent evaluation beat one long run.

3. Evaluation

I assembled a validation set of questions and compared answers side-by-side against the base model. The criterion that matters isn’t fluency — the base model is already fluent — but factual precision inside the domain and honesty at its edges: acknowledging what it doesn’t know is a positive result, not a failure.

4. The RAG layer

With the model already specialized, RAG grounds answers in the document base. This is where combining both techniques pays off: a model fluent in the domain’s vocabulary interprets retrieved passages better, and makes fewer mistakes synthesizing them.

from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "Refauna/tapirus-open-4b"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map="auto")

# Os trechos recuperados entram como contexto, não como conhecimento implícito.
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Contexto:\n{retrieved_chunks}\n\nPergunta: {question}"},
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

output = model.generate(inputs, max_new_tokens=512)

Why publish with open weights

Publishing on Hugging Face isn’t about distribution, it’s about scrutiny. A domain model that can’t be audited by people who know the domain has limited value. The model card matters as much as the weights: it has to state the scope, the source data, the known limitations and — above all — what the model should not be used to decide.

A model specialized in conservation does not replace technical field assessment. It speeds up reading, synthesis and hypothesis framing. Making that explicit is part of shipping it.

What I’m taking to the next SLM

1. A narrow scope is an advantage, not a limitation. The sharper the boundary, the better the result per unit of effort.
2. The base model’s license is a product decision. It determines what you can publish at the end — and that needs checking before the first training run, not after.
3. Data quality and consistency beat volume. A few well-formatted examples outperform many irregular ones.
4. Always evaluate against the base model.Without that comparison you can’t tell whether fine tuning helped or just changed the style of the answer.
5. Fine tuning and RAG are layers, not alternatives. One solves form, the other solves fact.

Tapirus Open 4B is still evolving. If you work with ecological restoration, with small models, or with both, I’m genuinely interested in criticism — especially the technical kind. I’m around on LinkedIn.